[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:39 dist-test-slave-6vgd krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:39 dist-test-slave-6vgd krb5kdc[399](info): set up 2 sockets
Sep 28 23:11:39 dist-test-slave-6vgd krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 23:11:40.044976 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:40.151546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:40.227584 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:40.227777 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:40.227805 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:40.228909 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:40.228947 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:40.228976 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:40.228992 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:40.229012 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:40.230476 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:40.230726 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:40.230945 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:11:40.235749 412 server_base.cc:1034] running on GCE node
W20260928 23:11:40.235837 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.235759 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.235759 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:40.236207 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:40.236480 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:40.237612 412 hybrid_clock.cc:648] HybridClock initialized: now 1790637100237602 us; error 27 us; skew 500 ppm
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:40.239789 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:40.240254 412 webserver.cc:460] Webserver started at http://127.0.98.62:38273/ using document root <none> and password file <none>
I20260928 23:11:40.240443 412 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:40.240489 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:40.240578 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:40.241585 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "5ecce41f4ce142e8826c09692ca73776"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "6e35a6fa26b80b405438f0476d5037a6"
server_key_iv: "f9b7285c4f4b5f9ea03a7234131e4311"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.241885 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ecce41f4ce142e8826c09692ca73776"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "6e35a6fa26b80b405438f0476d5037a6"
server_key_iv: "f9b7285c4f4b5f9ea03a7234131e4311"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.243040 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:11:40.243629 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.243777 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:40.243854 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "5ecce41f4ce142e8826c09692ca73776"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "6e35a6fa26b80b405438f0476d5037a6"
server_key_iv: "f9b7285c4f4b5f9ea03a7234131e4311"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.243920 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:40.258344 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:40.259918 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:40.260053 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:40.263676 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40335
I20260928 23:11:40.263705 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40335 every 8 connection(s)
I20260928 23:11:40.264004 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:40.264798 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:40.265301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 412
I20260928 23:11:40.265407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:40.266158 392 external_mini_cluster.cc:1428] Setting key 441f8cd00c92216a7e12da6d477a1d8c
I20260928 23:11:40.267715 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776: Bootstrap starting.
I20260928 23:11:40.268357 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:40.268673 480 log.cc:824] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:40.269410 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776: No bootstrap required, opened a new log
I20260928 23:11:40.271512 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260928 23:11:40.271675 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:40.271713 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ecce41f4ce142e8826c09692ca73776, State: Initialized, Role: FOLLOWER
I20260928 23:11:40.271835 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260928 23:11:40.271910 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:40.271945 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:40.271989 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:40.272506 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260928 23:11:40.272588 480 leader_election.cc:302] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ecce41f4ce142e8826c09692ca73776; no voters:
I20260928 23:11:40.272738 480 leader_election.cc:288] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:40.272766 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:40.272899 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [term 1 LEADER]: Becoming Leader. State: Replica: 5ecce41f4ce142e8826c09692ca73776, State: Running, Role: LEADER
I20260928 23:11:40.272994 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:40.273031 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } }
I20260928 23:11:40.273392 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ecce41f4ce142e8826c09692ca73776. Latest consensus state: current_term: 1 leader_uuid: "5ecce41f4ce142e8826c09692ca73776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260928 23:11:40.273399 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ecce41f4ce142e8826c09692ca73776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ecce41f4ce142e8826c09692ca73776" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40335 } } }
I20260928 23:11:40.273557 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:40.273566 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:40.273789 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:40.274289 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:40.275909 494 catalog_manager.cc:1347] Generated new cluster ID: 05f1a1eb258649be85c227282a323b04
I20260928 23:11:40.275971 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:40.283834 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.267019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.0.98.62:40335)
0928 23:11:40.267206 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.267215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.267399 (+ 184us) server_negotiation.cc:407] Connection header received
0928 23:11:40.269894 (+ 2495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.269900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.269951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.270060 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.272100 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.273599 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.275025 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.275572 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.275674 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.279590 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.279594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.279594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.279604 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.282013 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.282476 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.282478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.282479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.282569 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.282823 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.282823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.282824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.283086 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.283231 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.283502 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.283625 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260928 23:11:40.300523 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:40.301023 494 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:40.304205 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ecce41f4ce142e8826c09692ca73776: Generated new TSK 0
I20260928 23:11:40.304358 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:40.334340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:40.411645 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:40.411830 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:40.411850 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:40.412921 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:40.412963 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:40.412993 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:40.414376 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:40.414592 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:40.414791 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:40.419371 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.419427 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.419370 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:40.419497 506 server_base.cc:1034] running on GCE node
I20260928 23:11:40.419772 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:40.420006 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:40.421142 506 hybrid_clock.cc:648] HybridClock initialized: now 1790637100421134 us; error 27 us; skew 500 ppm
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:40.423429 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:40.423822 506 webserver.cc:460] Webserver started at http://127.0.98.1:45543/ using document root <none> and password file <none>
I20260928 23:11:40.424016 506 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:40.424062 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:40.424180 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:40.425134 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f5b71bed7834509bb07a390602bb0d9"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "50cd5d543fbe1704652e3f059c0f2295"
server_key_iv: "af1c43612108bd8a33b18c33e23e5f15"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.425496 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f5b71bed7834509bb07a390602bb0d9"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "50cd5d543fbe1704652e3f059c0f2295"
server_key_iv: "af1c43612108bd8a33b18c33e23e5f15"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.426633 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:40.427232 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.427366 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:40.427445 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "4f5b71bed7834509bb07a390602bb0d9"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "50cd5d543fbe1704652e3f059c0f2295"
server_key_iv: "af1c43612108bd8a33b18c33e23e5f15"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.427513 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:40.436580 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:40.438234 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:40.438356 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:40.438555 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:40.438819 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:40.438851 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.438879 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:40.438902 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.444522 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32841
I20260928 23:11:40.444562 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32841 every 8 connection(s)
I20260928 23:11:40.444847 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:11:40.448602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 506
I20260928 23:11:40.448684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:40.448817 392 external_mini_cluster.cc:1428] Setting key 7ae7777e15943d2e4f04152fb62508bf
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:40.453752 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.445533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55793 (local address 127.0.98.62:40335)
0928 23:11:40.445600 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.445603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.445744 (+ 141us) server_negotiation.cc:407] Connection header received
0928 23:11:40.446227 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.446228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.446240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.446291 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.447186 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.447685 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.448671 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.448963 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.449030 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.451379 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.451380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.451381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.451387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.452508 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.452853 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.452854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.452854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.452906 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.453118 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.453118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.453119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.453268 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.453323 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.453576 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.453663 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:11:40.453894 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.445481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.1:55793)
0928 23:11:40.445640 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:40.445676 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:40.446139 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:40.446372 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:40.446375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:40.446915 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:40.447100 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.447104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.447762 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.447763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.448595 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.448597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.449048 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.449049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.449181 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.449670 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:40.449679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:40.451295 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:40.452579 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.452585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.452587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.452790 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.452958 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.452958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.452959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.453064 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.453330 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:40.453336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:40.453464 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 23:11:40.453550 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:11:40.453695 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260928 23:11:40.454298 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260928 23:11:40.454380 635 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:40.454560 635 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260928 23:11:40.455109 444 ts_manager.cc:194] Registered new tserver with Master: 4f5b71bed7834509bb07a390602bb0d9 (127.0.98.1:32841)
I20260928 23:11:40.456008 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55793
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:40.472113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:40.549405 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:40.549597 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:40.549618 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:40.550698 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:40.550732 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:40.550756 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:40.552086 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:40.552251 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:40.552423 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:11:40.556938 640 server_base.cc:1034] running on GCE node
W20260928 23:11:40.557032 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.556972 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.556957 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:40.557404 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:40.557601 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:40.558739 640 hybrid_clock.cc:648] HybridClock initialized: now 1790637100558729 us; error 28 us; skew 500 ppm
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:40.560911 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:40.561324 640 webserver.cc:460] Webserver started at http://127.0.98.2:45239/ using document root <none> and password file <none>
I20260928 23:11:40.561518 640 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:40.561561 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:40.561666 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:40.562654 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "c20f3184852b4af19caedb8f87c3256a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "843870b3a83275e9bd30a18151fc5866"
server_key_iv: "59f7915e3d39c1e1c876ccbb68c8962e"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.563067 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c20f3184852b4af19caedb8f87c3256a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "843870b3a83275e9bd30a18151fc5866"
server_key_iv: "59f7915e3d39c1e1c876ccbb68c8962e"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.564477 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:40.565099 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.565265 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:40.565344 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "c20f3184852b4af19caedb8f87c3256a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "843870b3a83275e9bd30a18151fc5866"
server_key_iv: "59f7915e3d39c1e1c876ccbb68c8962e"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.565403 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:40.572065 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:40.573874 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:40.573993 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:40.574203 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:40.574429 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:40.574457 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.574476 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:40.574492 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.580202 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40839
I20260928 23:11:40.580236 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40839 every 8 connection(s)
I20260928 23:11:40.580518 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:40.586174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 640
I20260928 23:11:40.586272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:40.586438 392 external_mini_cluster.cc:1428] Setting key ae125a9982185fc3971a8bab7bd6724c
I20260928 23:11:40.588727 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.581088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48407 (local address 127.0.98.62:40335)
0928 23:11:40.581159 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.581161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.581315 (+ 154us) server_negotiation.cc:407] Connection header received
0928 23:11:40.581784 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.581785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.581795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.581842 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.582674 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.583150 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.584028 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.584202 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.584243 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.586097 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.586099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.586100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.586107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.587483 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.587824 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.587825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.587826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.587882 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.588162 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.588162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.588162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.588286 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.588346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.588562 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.588640 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:11:40.588793 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.581066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.2:48407)
0928 23:11:40.581185 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:40.581243 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:40.581707 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:40.581895 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:40.581897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:40.582429 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:40.582608 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.582613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.583206 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.583207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.583971 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.583973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.584251 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.584251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.584338 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.584739 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:40.584747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:40.585998 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:40.587555 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.587562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.587564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.587761 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.587953 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.587954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.587955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.588099 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.588340 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:40.588343 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:40.588448 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:11:40.588529 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 23:11:40.588641 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260928 23:11:40.589169 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260928 23:11:40.589279 769 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:40.589459 769 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260928 23:11:40.589802 444 ts_manager.cc:194] Registered new tserver with Master: c20f3184852b4af19caedb8f87c3256a (127.0.98.2:40839)
I20260928 23:11:40.590161 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48407
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:40.608317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:40.685390 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:40.685585 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:40.685606 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:40.686662 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:40.686692 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:40.686717 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:40.688060 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40335
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:40.688230 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:40.688395 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:40.693159 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:40.693322 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:40.693363 774 server_base.cc:1034] running on GCE node
W20260928 23:11:40.693683 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:40.693874 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:40.694108 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:40.695241 774 hybrid_clock.cc:648] HybridClock initialized: now 1790637100695227 us; error 28 us; skew 500 ppm
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:40.697433 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:40.697850 774 webserver.cc:460] Webserver started at http://127.0.98.3:46009/ using document root <none> and password file <none>
I20260928 23:11:40.698040 774 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:40.698086 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:40.698207 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:40.699157 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "4287e036cbdd4940a1492824def06c7a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "8869dab17ae8b84000396f686c69605a"
server_key_iv: "c2104a9f531a613df7c46d57edbcadbb"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.699468 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4287e036cbdd4940a1492824def06c7a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "8869dab17ae8b84000396f686c69605a"
server_key_iv: "c2104a9f531a613df7c46d57edbcadbb"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.700626 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:40.701200 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.701368 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.701443 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "4287e036cbdd4940a1492824def06c7a"
format_stamp: "Formatted at 2026-09-28 23:11:40 on dist-test-slave-6vgd"
server_key: "8869dab17ae8b84000396f686c69605a"
server_key_iv: "c2104a9f531a613df7c46d57edbcadbb"
server_key_version: "encryptionkey@0"
I20260928 23:11:40.701505 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:40.714188 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:40.715708 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:40.715832 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:40.716018 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:40.716284 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:40.716315 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.716341 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:40.716358 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:40.722155 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36807
I20260928 23:11:40.722218 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36807 every 8 connection(s)
I20260928 23:11:40.722466 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:40.730109 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.723072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47701 (local address 127.0.98.62:40335)
0928 23:11:40.723115 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.723116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.723286 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:11:40.723722 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.723723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.723731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.723781 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.724589 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.725069 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.725986 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.726164 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.726213 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.727985 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.727987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.727987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.727994 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.729035 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.729400 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.729401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.729401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.729447 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.729617 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.729617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.729617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.729709 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.729752 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.729952 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.730025 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:11:40.730311 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.723038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40335 (local address 127.0.98.3:47701)
0928 23:11:40.723178 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:40.723216 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:40.723651 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:40.723831 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:40.723834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:40.724341 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:40.724527 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.724531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.725126 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.725127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.725922 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:40.725924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.726220 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.726221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:40.726307 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.726675 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:40.726682 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:40.727895 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:40.729098 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.729104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.729107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.729350 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.729492 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:40.729493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:40.729493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:40.729583 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:40.729748 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:40.729750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:40.729849 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 23:11:40.729943 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:11:40.730072 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 23:11:40.730720 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40335
I20260928 23:11:40.730803 903 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:40.730935 903 heartbeater.cc:511] Master 127.0.98.62:40335 requested a full tablet report, sending...
I20260928 23:11:40.731263 444 ts_manager.cc:194] Registered new tserver with Master: 4287e036cbdd4940a1492824def06c7a (127.0.98.3:36807)
I20260928 23:11:40.731596 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47701
I20260928 23:11:40.732227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 774
I20260928 23:11:40.732308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:40.732452 392 external_mini_cluster.cc:1428] Setting key a243f09b50c2926a2a13454246434a70
I20260928 23:11:40.733345 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:11:40.753840 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.746774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.0.98.1:32841)
0928 23:11:40.746937 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.746958 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.746969 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:11:40.746993 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.746995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.747007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.747081 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.747963 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.748668 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.749364 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.749678 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.749727 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.751667 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.751674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.751675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.751681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.752903 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.753176 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.753179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.753179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.753275 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.753407 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.753408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.753408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.753534 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.753636 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.753704 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.753763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
W20260928 23:11:40.754189 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51848
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:11:40.776535 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:40.770527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.0.98.1:32841)
0928 23:11:40.770580 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:40.770583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:40.770638 (+ 55us) server_negotiation.cc:407] Connection header received
0928 23:11:40.770689 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:40.770689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:40.770699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:40.770750 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:40.771229 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.771710 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.772406 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:40.772607 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:40.772656 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:40.774689 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:40.774691 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:40.774692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:40.774699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:40.775750 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.776026 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.776027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.776027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.776070 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:40.776193 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:40.776193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:40.776193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:40.776270 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:40.776348 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:40.776407 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 23:11:40.776464 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:11:40.776995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 506
I20260928 23:11:40.781001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 640
I20260928 23:11:40.785615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 774
I20260928 23:11:40.789860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 412
2026-09-28T23:11:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1687 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[917](info): set up 2 sockets
Sep 28 23:11:40 dist-test-slave-6vgd krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 23:11:41.732491 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:41 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:41.838088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:41.914628 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:41.914830 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:41.914861 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:41.915957 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:41.915998 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:41.916018 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:41.916033 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:41.916047 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:41.917490 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:41.917725 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:41.917920 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:41.922474 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:41.922497 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:41.922482 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:41.922511 930 server_base.cc:1034] running on GCE node
I20260928 23:11:41.922840 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:41.923108 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:41.924242 930 hybrid_clock.cc:648] HybridClock initialized: now 1790637101924228 us; error 30 us; skew 500 ppm
Sep 28 23:11:41 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:41.926462 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:41.926846 930 webserver.cc:460] Webserver started at http://127.0.98.62:45683/ using document root <none> and password file <none>
I20260928 23:11:41.927037 930 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:41.927078 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:41.927191 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:41.928160 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "2e25fd8dfb3745b08feb5fe36680e64d"
format_stamp: "Formatted at 2026-09-28 23:11:41 on dist-test-slave-6vgd"
server_key: "e302a2527d4b0d2f6f3dc2ddcee93465"
server_key_iv: "d5e60643318686638f2be05e3733c6e4"
server_key_version: "encryptionkey@0"
I20260928 23:11:41.928467 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e25fd8dfb3745b08feb5fe36680e64d"
format_stamp: "Formatted at 2026-09-28 23:11:41 on dist-test-slave-6vgd"
server_key: "e302a2527d4b0d2f6f3dc2ddcee93465"
server_key_iv: "d5e60643318686638f2be05e3733c6e4"
server_key_version: "encryptionkey@0"
I20260928 23:11:41.929682 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 23:11:41.930307 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:41.930436 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:41.930516 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "2e25fd8dfb3745b08feb5fe36680e64d"
format_stamp: "Formatted at 2026-09-28 23:11:41 on dist-test-slave-6vgd"
server_key: "e302a2527d4b0d2f6f3dc2ddcee93465"
server_key_iv: "d5e60643318686638f2be05e3733c6e4"
server_key_version: "encryptionkey@0"
I20260928 23:11:41.930577 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:41.937999 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:41.939555 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:41.939673 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:41.943137 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35727
I20260928 23:11:41.943166 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35727 every 8 connection(s)
I20260928 23:11:41.943477 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:41.944253 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:41.946367 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d: Bootstrap starting.
I20260928 23:11:41.946880 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:41.947139 998 log.cc:824] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:41.947713 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d: No bootstrap required, opened a new log
I20260928 23:11:41.949260 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260928 23:11:41.949389 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:41.949421 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e25fd8dfb3745b08feb5fe36680e64d, State: Initialized, Role: FOLLOWER
I20260928 23:11:41.949553 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260928 23:11:41.949608 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:41.949640 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:41.949682 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:41.950140 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260928 23:11:41.950223 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e25fd8dfb3745b08feb5fe36680e64d; no voters:
I20260928 23:11:41.950374 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:41.950402 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:41.950619 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:41.950687 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [term 1 LEADER]: Becoming Leader. State: Replica: 2e25fd8dfb3745b08feb5fe36680e64d, State: Running, Role: LEADER
I20260928 23:11:41.950775 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } }
I20260928 23:11:41.951088 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } } }
I20260928 23:11:41.951160 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:41.951328 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e25fd8dfb3745b08feb5fe36680e64d. Latest consensus state: current_term: 1 leader_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e25fd8dfb3745b08feb5fe36680e64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35727 } } }
I20260928 23:11:41.951387 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:41.951370 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:41.951649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 930
I20260928 23:11:41.951741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:41.951884 392 external_mini_cluster.cc:1428] Setting key c9288878576127054517e8f7e4c31e4f
I20260928 23:11:41.951927 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:41.954039 1010 catalog_manager.cc:1347] Generated new cluster ID: 547db1ee54944cf38c994b050d9ec181
I20260928 23:11:41.954087 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:11:41 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:41.961867 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:41.962327 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:41.963083 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:41.952494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.0.98.62:35727)
0928 23:11:41.952709 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:41.952718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:41.952746 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:11:41.952830 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:41.952835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:41.952875 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:41.952989 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:41.953708 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:41.955100 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:41.956099 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:41.956721 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:41.956824 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:41.959454 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:41.959459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:41.959459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:41.959471 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:41.961306 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:41.961686 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:41.961688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:41.961689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:41.961796 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:41.962001 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:41.962002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:41.962003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:41.962286 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:41.962444 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:41.962598 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 23:11:41.962852 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260928 23:11:41.974263 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e25fd8dfb3745b08feb5fe36680e64d: Generated new TSK 0
I20260928 23:11:41.974440 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:42.007220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:42.094000 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:42.094221 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:42.094245 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:42.095369 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:42.095409 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:42.095438 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:42.096813 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:42.097023 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:42.097265 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:42.101979 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:42.102005 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:42.102236 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:42.102267 1024 server_base.cc:1034] running on GCE node
I20260928 23:11:42.102463 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:42.102708 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:42.103869 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790637102103852 us; error 33 us; skew 500 ppm
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:42.106222 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:42.106596 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46073/ using document root <none> and password file <none>
I20260928 23:11:42.106784 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:42.106832 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:42.106946 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:42.107940 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "bcf9487df44440038f119f5fb2b0b737"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "fe87603d60ab80a8dcac6923fea9026d"
server_key_iv: "80cf336754bae00d43ded9c9580b608d"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.108260 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bcf9487df44440038f119f5fb2b0b737"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "fe87603d60ab80a8dcac6923fea9026d"
server_key_iv: "80cf336754bae00d43ded9c9580b608d"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.109436 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:42.110033 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.110172 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:42.110257 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "bcf9487df44440038f119f5fb2b0b737"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "fe87603d60ab80a8dcac6923fea9026d"
server_key_iv: "80cf336754bae00d43ded9c9580b608d"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.110314 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:42.121570 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:42.123114 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:42.123245 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:42.123428 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:42.123682 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:42.123715 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.123744 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:42.123767 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.130352 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43733
I20260928 23:11:42.130431 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43733 every 8 connection(s)
I20260928 23:11:42.130749 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:11:42.131861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1024
I20260928 23:11:42.131963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:42.132117 392 external_mini_cluster.cc:1428] Setting key d4ad4a174a81aa82f6864309d4832847
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:11:42.140136 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.131535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57601 (local address 127.0.98.62:35727)
0928 23:11:42.131577 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.131579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.131841 (+ 262us) server_negotiation.cc:407] Connection header received
0928 23:11:42.132512 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.132513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.132526 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.132562 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.133765 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.134350 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.135302 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.135518 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.135579 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.137709 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.137710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.137710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.137716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.138972 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.139425 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.139426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.139426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.139476 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.139676 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.139676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.139677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.139766 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.139809 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.140004 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.140068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:11:42.140374 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.131524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35727 (local address 127.0.98.1:57601)
0928 23:11:42.131671 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.131727 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.132390 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.132785 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.132789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.133463 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:42.133651 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.133656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.134427 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.134428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.135225 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.135227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.135637 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.135638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.135739 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.136242 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:42.136249 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:42.137617 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:42.139051 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.139057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.139060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.139350 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.139533 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.139534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.139534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.139636 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.139808 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:42.139811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:42.139918 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.139992 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.140106 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260928 23:11:42.140808 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35727
I20260928 23:11:42.140897 1153 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:42.141088 1153 heartbeater.cc:511] Master 127.0.98.62:35727 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 23:11:42.141601 962 ts_manager.cc:194] Registered new tserver with Master: bcf9487df44440038f119f5fb2b0b737 (127.0.98.1:43733)
I20260928 23:11:42.142243 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:42.153978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:42.231880 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:42.232070 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:42.232092 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:42.233198 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:42.233260 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:42.233294 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:42.234668 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:42.234836 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:42.234999 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:42.239562 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:42.239552 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:42.239681 1158 server_base.cc:1034] running on GCE node
W20260928 23:11:42.239552 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:42.239926 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:42.240114 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:42.241267 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790637102241238 us; error 43 us; skew 500 ppm
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:42.243438 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:42.243909 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35669/ using document root <none> and password file <none>
I20260928 23:11:42.244107 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:42.244158 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:42.244279 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:42.245337 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d85917c840848d5ac06eeedc726b6ab"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "58dff4b11f8a265010eefc7c1802f37d"
server_key_iv: "4f0e5800c40abfa5bf0feb5118b191f2"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.245654 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d85917c840848d5ac06eeedc726b6ab"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "58dff4b11f8a265010eefc7c1802f37d"
server_key_iv: "4f0e5800c40abfa5bf0feb5118b191f2"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.246781 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:42.247395 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.247535 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:42.247607 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "3d85917c840848d5ac06eeedc726b6ab"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "58dff4b11f8a265010eefc7c1802f37d"
server_key_iv: "4f0e5800c40abfa5bf0feb5118b191f2"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.247671 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:42.270180 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:42.271914 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:42.272037 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:42.272249 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:42.272539 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:42.272571 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.272593 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:42.272615 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.277866 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41645
I20260928 23:11:42.277923 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41645 every 8 connection(s)
I20260928 23:11:42.278208 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:42.286343 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.278840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52199 (local address 127.0.98.62:35727)
0928 23:11:42.278889 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.278891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.279065 (+ 174us) server_negotiation.cc:407] Connection header received
0928 23:11:42.279561 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.279562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.279578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.279624 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.280525 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.281022 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.282033 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.282217 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.282272 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.284159 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.284160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.284161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.284168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.285182 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.285535 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.285536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.285537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.285586 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.285786 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.285787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.285788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.285888 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.285942 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.286180 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.286256 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:11:42.286507 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.278801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35727 (local address 127.0.98.2:52199)
0928 23:11:42.278958 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.278993 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.279472 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.279703 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.279708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.280249 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:42.280436 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.280440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.281101 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.281102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.281948 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.281951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.282282 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.282283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.282406 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.282848 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:42.282855 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:42.284072 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:42.285271 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.285277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.285279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.285473 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.285638 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.285638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.285638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.285733 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.285940 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:42.285946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:42.286070 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.286155 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.286307 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260928 23:11:42.286883 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35727
I20260928 23:11:42.286965 1287 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:42.287122 1287 heartbeater.cc:511] Master 127.0.98.62:35727 requested a full tablet report, sending...
I20260928 23:11:42.287443 962 ts_manager.cc:194] Registered new tserver with Master: 3d85917c840848d5ac06eeedc726b6ab (127.0.98.2:41645)
I20260928 23:11:42.287846 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52199
I20260928 23:11:42.287850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1158
I20260928 23:11:42.287931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:42.288094 392 external_mini_cluster.cc:1428] Setting key 72f5de9b35a00c7a3ac4d6563228d957
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:42.309556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:42.386125 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:42.386319 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:42.386349 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:42.387487 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:42.387526 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:42.387559 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:42.388947 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35727
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:42.389170 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:42.389385 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:42.393865 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:42.393982 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:42.393886 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:42.393980 1292 server_base.cc:1034] running on GCE node
I20260928 23:11:42.394285 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:42.394469 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:42.395594 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790637102395584 us; error 32 us; skew 500 ppm
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:42.400879 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:42.401304 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39397/ using document root <none> and password file <none>
I20260928 23:11:42.401503 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:42.401546 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:42.401664 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:42.402665 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "45d1bd441b3d4df394af73c7284747b0"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "39c980f53bf22e4b0e44bb03a902a2ff"
server_key_iv: "5aaa14b05d2bf895756f4620b59da6bb"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.403019 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45d1bd441b3d4df394af73c7284747b0"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "39c980f53bf22e4b0e44bb03a902a2ff"
server_key_iv: "5aaa14b05d2bf895756f4620b59da6bb"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.404163 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:42.404763 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.404892 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:42.404968 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "45d1bd441b3d4df394af73c7284747b0"
format_stamp: "Formatted at 2026-09-28 23:11:42 on dist-test-slave-6vgd"
server_key: "39c980f53bf22e4b0e44bb03a902a2ff"
server_key_iv: "5aaa14b05d2bf895756f4620b59da6bb"
server_key_version: "encryptionkey@0"
I20260928 23:11:42.405025 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:42.413774 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:42.415393 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:42.415521 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:42.415704 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:42.415983 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:42.416021 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.416047 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:42.416070 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:42.421294 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41073
I20260928 23:11:42.421331 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41073 every 8 connection(s)
I20260928 23:11:42.421613 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:11:42.423244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1292
I20260928 23:11:42.423323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:42.423454 392 external_mini_cluster.cc:1428] Setting key 13e3aadf11d80461246e9129832888d5
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:42.429672 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.422225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48523 (local address 127.0.98.62:35727)
0928 23:11:42.422269 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.422270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.422457 (+ 187us) server_negotiation.cc:407] Connection header received
0928 23:11:42.422935 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.422936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.422948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.422994 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.423990 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.424477 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.425465 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.425650 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.425703 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.427509 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.427510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.427511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.427518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.428496 (+ 978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.428830 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.428831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.428831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.428879 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.429083 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.429084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.429084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.429185 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.429272 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.429511 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.429587 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:11:42.429822 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.422194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35727 (local address 127.0.98.3:48523)
0928 23:11:42.422347 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.422381 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.422845 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.423074 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.423077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.423686 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:42.423910 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.423916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.424552 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.424553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.425393 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.425395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.425711 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.425713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.425835 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.426212 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:42.426219 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:42.427428 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:42.428562 (+ 1134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.428568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.428571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.428768 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.428929 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:42.428930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:42.428931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:42.429030 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:42.429263 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:42.429268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:42.429398 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.429482 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.429621 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260928 23:11:42.430202 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35727
I20260928 23:11:42.430297 1421 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:42.430457 1421 heartbeater.cc:511] Master 127.0.98.62:35727 requested a full tablet report, sending...
I20260928 23:11:42.430795 962 ts_manager.cc:194] Registered new tserver with Master: 45d1bd441b3d4df394af73c7284747b0 (127.0.98.3:41073)
I20260928 23:11:42.431141 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48523
I20260928 23:11:42.434810 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:11:42.441354 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.436138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.0.98.62:35727)
0928 23:11:42.436194 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.436196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.436286 (+ 90us) server_negotiation.cc:407] Connection header received
0928 23:11:42.436344 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.436345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.436355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.436401 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.436916 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.437423 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.438104 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.438289 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.438341 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.439342 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.439343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.439344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.439350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.440337 (+ 987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.440642 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.440643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.440644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.440693 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.440848 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.440849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.440849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.440980 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.441064 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.441179 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.441269 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:11:42.443084 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:42.444639 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:42.450415 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41645 (local address 127.0.0.1:54524)
0928 23:11:42.446732 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.446771 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.446832 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.447069 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.447072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.447074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:42.447138 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.447140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448370 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.448371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.449446 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.449448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450097 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.450213 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450226 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.450296 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.450346 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260928 23:11:42.450642 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43733 (local address 127.0.0.1:48480)
0928 23:11:42.446918 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.446926 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.446968 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.447212 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.447212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.447214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:42.447259 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.447260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448123 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.448124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.449411 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.449414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450403 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.450464 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450468 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.450517 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.450571 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
I20260928 23:11:42.450690 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.0.98.2:41645)
0928 23:11:42.446853 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.446857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.446866 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:42.446904 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.446906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.446920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.446989 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:42.447198 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448281 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.449524 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450026 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450089 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450179 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.450316 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.450357 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":64,"threads_started":1}
I20260928 23:11:42.450755 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41073 (local address 127.0.0.1:51252)
0928 23:11:42.446872 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.446882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.446940 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.447086 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.447087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.447089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:42.447155 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.447157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448596 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.448597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.449371 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.449373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450581 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.450640 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450643 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.450671 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.450702 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260928 23:11:42.450868 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.0.98.1:43733)
0928 23:11:42.446866 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.446870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.447020 (+ 150us) server_negotiation.cc:407] Connection header received
0928 23:11:42.447061 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.447063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.447077 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.447140 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:42.447321 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448038 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.449482 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450253 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450369 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.450749 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.450794 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260928 23:11:42.451079 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.446507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.0.98.3:41073)
0928 23:11:42.446768 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.446773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.446935 (+ 162us) server_negotiation.cc:407] Connection header received
0928 23:11:42.447001 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.447003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.447018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.447085 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:42.447236 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.448510 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.449441 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.450477 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.450889 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.450953 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.450979 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.451014 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260928 23:11:42.451377 1221 tablet_service.cc:1467] Processing CreateTablet for tablet f767e294e216472dbd8c19fc27af8231 (DEFAULT_TABLE table=test-table [id=b2b393c14b164d9ca4b3db3bb7a025a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:42.451447 1078 tablet_service.cc:1467] Processing CreateTablet for tablet f767e294e216472dbd8c19fc27af8231 (DEFAULT_TABLE table=test-table [id=b2b393c14b164d9ca4b3db3bb7a025a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:42.451758 1078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767e294e216472dbd8c19fc27af8231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:42.451771 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767e294e216472dbd8c19fc27af8231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:42.451722 1355 tablet_service.cc:1467] Processing CreateTablet for tablet f767e294e216472dbd8c19fc27af8231 (DEFAULT_TABLE table=test-table [id=b2b393c14b164d9ca4b3db3bb7a025a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:42.451998 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f767e294e216472dbd8c19fc27af8231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:42.454463 1435 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Bootstrap starting.
I20260928 23:11:42.454906 1436 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Bootstrap starting.
I20260928 23:11:42.455051 1437 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Bootstrap starting.
I20260928 23:11:42.455526 1435 tablet_bootstrap.cc:654] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:42.455526 1436 tablet_bootstrap.cc:654] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:42.455549 1437 tablet_bootstrap.cc:654] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:42.456005 1437 log.cc:824] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:42.456149 1436 log.cc:824] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:42.456542 1435 log.cc:824] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:42.456676 1437 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: No bootstrap required, opened a new log
I20260928 23:11:42.456739 1437 ts_tablet_manager.cc:1397] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:42.456861 1436 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: No bootstrap required, opened a new log
I20260928 23:11:42.456912 1436 ts_tablet_manager.cc:1397] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:11:42.457788 1435 tablet_bootstrap.cc:492] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: No bootstrap required, opened a new log
I20260928 23:11:42.457854 1435 ts_tablet_manager.cc:1397] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:11:42.458259 1436 raft_consensus.cc:348] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.458410 1436 raft_consensus.cc:374] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:42.458451 1436 raft_consensus.cc:729] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d85917c840848d5ac06eeedc726b6ab, State: Initialized, Role: FOLLOWER
I20260928 23:11:42.458408 1437 raft_consensus.cc:348] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.458560 1437 raft_consensus.cc:374] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:42.458593 1437 raft_consensus.cc:729] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45d1bd441b3d4df394af73c7284747b0, State: Initialized, Role: FOLLOWER
I20260928 23:11:42.458575 1436 consensus_queue.cc:260] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.458700 1437 consensus_queue.cc:260] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.458834 1436 ts_tablet_manager.cc:1428] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:11:42.458834 1287 heartbeater.cc:503] Master 127.0.98.62:35727 was elected leader, sending a full tablet report...
I20260928 23:11:42.458927 1437 ts_tablet_manager.cc:1428] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:11:42.459025 1421 heartbeater.cc:503] Master 127.0.98.62:35727 was elected leader, sending a full tablet report...
I20260928 23:11:42.459627 1435 raft_consensus.cc:348] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.459805 1435 raft_consensus.cc:374] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:42.459842 1435 raft_consensus.cc:729] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf9487df44440038f119f5fb2b0b737, State: Initialized, Role: FOLLOWER
I20260928 23:11:42.459945 1435 consensus_queue.cc:260] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.460145 1435 ts_tablet_manager.cc:1428] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:11:42.460199 1153 heartbeater.cc:503] Master 127.0.98.62:35727 was elected leader, sending a full tablet report...
W20260928 23:11:42.528728 1288 tablet.cc:2369] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:42.609294 1442 raft_consensus.cc:482] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:42.609426 1442 raft_consensus.cc:504] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.609774 1442 leader_election.cc:288] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d85917c840848d5ac06eeedc726b6ab (127.0.98.2:41645), bcf9487df44440038f119f5fb2b0b737 (127.0.98.1:43733)
I20260928 23:11:42.612469 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.609935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43733 (local address 127.0.98.3:41535)
0928 23:11:42.609996 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.610008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.610052 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.610187 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.610188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.610189 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:42.610258 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.610259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.610868 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.610869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.611642 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.611644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.612204 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.612205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.612290 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.612294 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.612339 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.612378 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":24}
I20260928 23:11:42.612483 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.609992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60839 (local address 127.0.98.2:41645)
0928 23:11:42.610038 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.610040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.610202 (+ 162us) server_negotiation.cc:407] Connection header received
0928 23:11:42.610262 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.610263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.610275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.610320 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:42.610446 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.610919 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.611814 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.612177 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.612254 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.612264 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.612381 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.612419 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:11:42.612483 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.610035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41535 (local address 127.0.98.1:43733)
0928 23:11:42.610085 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.610087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.610094 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:11:42.610121 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.610122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.610133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.610175 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:42.610315 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.610797 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.611727 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.612131 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.612195 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.612205 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.612367 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.612412 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:11:42.612474 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.609955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41645 (local address 127.0.98.3:60839)
0928 23:11:42.610132 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:42.610141 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:42.610209 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:42.610326 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:42.610327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:42.610328 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:42.610395 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.610396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.610988 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.610989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.611737 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:42.611738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.612267 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.612268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:42.612324 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.612326 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:11:42.612372 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:11:42.612399 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260928 23:11:42.612834 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767e294e216472dbd8c19fc27af8231" candidate_uuid: "45d1bd441b3d4df394af73c7284747b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9487df44440038f119f5fb2b0b737" is_pre_election: true
I20260928 23:11:42.612838 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767e294e216472dbd8c19fc27af8231" candidate_uuid: "45d1bd441b3d4df394af73c7284747b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d85917c840848d5ac06eeedc726b6ab" is_pre_election: true
I20260928 23:11:42.612969 1241 raft_consensus.cc:2454] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d1bd441b3d4df394af73c7284747b0 in term 0.
I20260928 23:11:42.612975 1107 raft_consensus.cc:2454] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45d1bd441b3d4df394af73c7284747b0 in term 0.
I20260928 23:11:42.613197 1311 leader_election.cc:302] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d1bd441b3d4df394af73c7284747b0, bcf9487df44440038f119f5fb2b0b737; no voters:
I20260928 23:11:42.613425 1442 raft_consensus.cc:2781] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:42.613469 1442 raft_consensus.cc:482] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:42.613483 1442 raft_consensus.cc:3037] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:42.614123 1442 raft_consensus.cc:504] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.614238 1442 leader_election.cc:288] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [CANDIDATE]: Term 1 election: Requested vote from peers 3d85917c840848d5ac06eeedc726b6ab (127.0.98.2:41645), bcf9487df44440038f119f5fb2b0b737 (127.0.98.1:43733)
I20260928 23:11:42.614383 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767e294e216472dbd8c19fc27af8231" candidate_uuid: "45d1bd441b3d4df394af73c7284747b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf9487df44440038f119f5fb2b0b737"
I20260928 23:11:42.614385 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f767e294e216472dbd8c19fc27af8231" candidate_uuid: "45d1bd441b3d4df394af73c7284747b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d85917c840848d5ac06eeedc726b6ab"
I20260928 23:11:42.614452 1241 raft_consensus.cc:3037] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:42.614455 1107 raft_consensus.cc:3037] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:42.615132 1107 raft_consensus.cc:2454] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d1bd441b3d4df394af73c7284747b0 in term 1.
I20260928 23:11:42.615307 1311 leader_election.cc:302] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45d1bd441b3d4df394af73c7284747b0, bcf9487df44440038f119f5fb2b0b737; no voters:
I20260928 23:11:42.615388 1442 raft_consensus.cc:2781] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:42.615535 1442 raft_consensus.cc:686] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [term 1 LEADER]: Becoming Leader. State: Replica: 45d1bd441b3d4df394af73c7284747b0, State: Running, Role: LEADER
I20260928 23:11:42.615638 1442 consensus_queue.cc:237] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } }
I20260928 23:11:42.615667 1241 raft_consensus.cc:2454] T f767e294e216472dbd8c19fc27af8231 P 3d85917c840848d5ac06eeedc726b6ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45d1bd441b3d4df394af73c7284747b0 in term 1.
I20260928 23:11:42.616479 961 catalog_manager.cc:5495] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45d1bd441b3d4df394af73c7284747b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "45d1bd441b3d4df394af73c7284747b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45d1bd441b3d4df394af73c7284747b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d85917c840848d5ac06eeedc726b6ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf9487df44440038f119f5fb2b0b737" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43733 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260928 23:11:42.631433 1154 tablet.cc:2369] T f767e294e216472dbd8c19fc27af8231 P bcf9487df44440038f119f5fb2b0b737: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:42.632634 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.626175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.0.98.1:43733)
0928 23:11:42.626230 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.626232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.626321 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:11:42.626383 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.626384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.626393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.626441 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.626934 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.627401 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.628078 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.628271 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.628324 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.630331 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.630342 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.630342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.630350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.631637 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.631937 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.631940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.631940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.631991 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.632146 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.632146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.632147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.632274 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.632398 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.632457 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.632548 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:11:42.653353 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.647303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.1:43733)
0928 23:11:42.647366 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.647368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.647414 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:11:42.647466 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.647467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.647476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.647521 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.647968 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.648426 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.649059 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.649271 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.649316 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.651352 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.651354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.651354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.651361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.652535 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.652801 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.652801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.652802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.652843 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.652959 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.652960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.652960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.653031 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.653091 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.653195 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.653287 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260928 23:11:42.672101 1422 tablet.cc:2369] T f767e294e216472dbd8c19fc27af8231 P 45d1bd441b3d4df394af73c7284747b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:42.672533 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:42.666559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.1:43733)
0928 23:11:42.666621 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:42.666623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:42.666663 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:11:42.666730 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:42.666731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:42.666741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:42.666788 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:42.667271 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.667739 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.668417 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:42.668597 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:42.668648 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:42.670614 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:42.670616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:42.670616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:42.670623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:42.671611 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.671891 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.671892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.671892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.671938 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:42.672092 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:42.672093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:42.672093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:42.672187 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:42.672279 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:42.672375 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:11:42.672443 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 23:11:42.672963 1059 tablet_service.cc:2908] Rejecting scan request for tablet f767e294e216472dbd8c19fc27af8231: Uninitialized: clean time has not yet been initialized
I20260928 23:11:42.673676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1024
I20260928 23:11:42.677752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1158
I20260928 23:11:42.681654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1292
I20260928 23:11:42.685179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 930
2026-09-28T23:11:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1896 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[1459](info): set up 2 sockets
Sep 28 23:11:42 dist-test-slave-6vgd krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260928 23:11:43.632428 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:43 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:43.738060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:43.814940 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:43.815143 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:43.815164 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:43.816289 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:43.816331 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:11:43.816344 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:43.816362 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:43.816386 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:43.816401 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:43.817896 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:43.818121 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:43.818320 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:43.822679 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:43.822710 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:43.822822 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:43.822870 1472 server_base.cc:1034] running on GCE node
I20260928 23:11:43.823084 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:43.823339 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:43.824476 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790637103824467 us; error 26 us; skew 500 ppm
Sep 28 23:11:43 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:43.826694 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:43.827098 1472 webserver.cc:460] Webserver started at http://127.0.98.62:41125/ using document root <none> and password file <none>
I20260928 23:11:43.827286 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:43.827332 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:43.827453 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:43.828430 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "87854f80d492429587659a6745f3ea3b"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "4325e6f1ee5ebfb2453e31a5c1e8245a"
server_key_iv: "5c359ff6c955549756413bc69d08a561"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.828739 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "87854f80d492429587659a6745f3ea3b"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "4325e6f1ee5ebfb2453e31a5c1e8245a"
server_key_iv: "5c359ff6c955549756413bc69d08a561"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.829958 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:43.830680 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:43.830853 1472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:43.830941 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "87854f80d492429587659a6745f3ea3b"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "4325e6f1ee5ebfb2453e31a5c1e8245a"
server_key_iv: "5c359ff6c955549756413bc69d08a561"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.830999 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:43.855671 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:43.857576 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:43.857730 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:43.861977 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39439
I20260928 23:11:43.862025 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39439 every 8 connection(s)
I20260928 23:11:43.862325 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:43.862920 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:43.865070 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Bootstrap starting.
I20260928 23:11:43.865645 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:43.865897 1550 log.cc:824] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:43.866533 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: No bootstrap required, opened a new log
I20260928 23:11:43.868470 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } }
I20260928 23:11:43.868609 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:43.868640 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87854f80d492429587659a6745f3ea3b, State: Initialized, Role: FOLLOWER
I20260928 23:11:43.868772 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } }
I20260928 23:11:43.868846 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:43.868877 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:43.868922 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:43.869449 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } }
I20260928 23:11:43.869544 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87854f80d492429587659a6745f3ea3b; no voters:
I20260928 23:11:43.869692 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:43.869784 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:43.869939 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [term 1 LEADER]: Becoming Leader. State: Replica: 87854f80d492429587659a6745f3ea3b, State: Running, Role: LEADER
I20260928 23:11:43.869925 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:43.870054 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } }
I20260928 23:11:43.870404 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87854f80d492429587659a6745f3ea3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } } }
I20260928 23:11:43.870486 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:43.870636 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87854f80d492429587659a6745f3ea3b. Latest consensus state: current_term: 1 leader_uuid: "87854f80d492429587659a6745f3ea3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87854f80d492429587659a6745f3ea3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39439 } } }
I20260928 23:11:43.870690 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:43.870977 1562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:43.871467 1562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:43.871912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1472
I20260928 23:11:43.872005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:43.872140 392 external_mini_cluster.cc:1428] Setting key 690fccdbc47495986f141b8febc20e70
I20260928 23:11:43.873459 1562 catalog_manager.cc:1347] Generated new cluster ID: e39c3280332d4588a7d2bc45cd4d4764
I20260928 23:11:43.873508 1562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:11:43.877303 1562 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:43.877766 1562 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 23:11:43 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:43.883500 1562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Generated new TSK 0
I20260928 23:11:43.883634 1562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:11:43.884092 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:43.872614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.0.98.62:39439)
0928 23:11:43.872785 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:43.872794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:43.872826 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:11:43.872917 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:43.872923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:43.872963 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:43.873070 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:43.873835 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:43.875251 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:43.876317 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:43.877439 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:43.877544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:43.880347 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:43.880352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:43.880353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:43.880365 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:43.882348 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:43.882731 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:43.882734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:43.882735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:43.882836 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:43.883045 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:43.883046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:43.883047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:43.883355 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:43.883478 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:43.883764 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 23:11:43.883926 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:43.905534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39439
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:43.982843 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:43.983042 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:43.983073 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:43.984230 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:43.984274 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:43.984302 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:43.984324 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:43.985857 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:43.986079 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:43.986275 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:43.990948 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:43.990928 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:43.990924 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:43.991273 1576 server_base.cc:1034] running on GCE node
I20260928 23:11:43.991456 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:43.991676 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:43.992905 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790637103992888 us; error 33 us; skew 500 ppm
Sep 28 23:11:43 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:43.995309 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:43.995697 1576 webserver.cc:460] Webserver started at http://127.0.98.1:35351/ using document root <none> and password file <none>
I20260928 23:11:43.995926 1576 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:43.995975 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:43.996102 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:43.997116 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "32097e0181bfb3ae5a08ebf203d6824a"
server_key_iv: "53c37bbbb0db7c94c72d7b77ac1993b1"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.997463 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "32097e0181bfb3ae5a08ebf203d6824a"
server_key_iv: "53c37bbbb0db7c94c72d7b77ac1993b1"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.998641 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:43.999266 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:43.999432 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:43.999516 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
format_stamp: "Formatted at 2026-09-28 23:11:43 on dist-test-slave-6vgd"
server_key: "32097e0181bfb3ae5a08ebf203d6824a"
server_key_iv: "53c37bbbb0db7c94c72d7b77ac1993b1"
server_key_version: "encryptionkey@0"
I20260928 23:11:43.999577 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:44.014869 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:44.016413 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:44.016541 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:44.016994 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:44.017032 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.017060 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:44.017077 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.023028 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36587
I20260928 23:11:44.023085 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36587 every 8 connection(s)
I20260928 23:11:44.023396 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:44.026051 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.017412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42223 (local address 127.0.98.62:39439)
0928 23:11:44.017462 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.017465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.017672 (+ 207us) server_negotiation.cc:407] Connection header received
0928 23:11:44.018153 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.018155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.018171 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.018248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.018992 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.019784 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.021247 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.021549 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.021626 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.023982 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.023984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.023984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.023991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.025038 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.025397 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.025397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.025398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.025443 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.025608 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.025609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.025609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.025696 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.025736 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.025932 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.025990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:11:44.026217 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.017407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.1:42223)
0928 23:11:44.017550 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.017588 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.018060 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.018302 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.018305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.018621 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.018894 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.018900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.019877 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.019879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.021128 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.021131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.021626 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.021627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.021720 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.022213 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.022224 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.023889 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.025097 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.025103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.025105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.025347 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.025487 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.025488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.025488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.025574 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.025734 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.025736 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.025838 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.025920 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.026028 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260928 23:11:44.026628 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39439
I20260928 23:11:44.026703 1707 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:44.026882 1707 heartbeater.cc:511] Master 127.0.98.62:39439 requested a full tablet report, sending...
I20260928 23:11:44.027395 1503 ts_manager.cc:194] Registered new tserver with Master: 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587)
I20260928 23:11:44.028139 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42223
I20260928 23:11:44.029865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1576
I20260928 23:11:44.029950 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:44.030129 392 external_mini_cluster.cc:1428] Setting key 1823542bab9599847022c1d829fca860
I20260928 23:11:44.032626 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.027646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43167 (local address 127.0.98.62:39439)
0928 23:11:44.027682 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.027684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.027732 (+ 48us) server_negotiation.cc:407] Connection header received
0928 23:11:44.027781 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.027782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.027790 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.027849 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.028077 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.028657 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.029394 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.029567 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.029615 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.030429 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.030431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.030431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.030436 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.031683 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.031966 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.031967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.031967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.032018 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.032173 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.032174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.032175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.032279 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.032335 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.032477 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.032540 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:11:44.032639 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.027582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.1:43167)
0928 23:11:44.027684 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.027693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.027742 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.027852 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.027852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.027977 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.028027 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.028028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.028743 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.028744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.029323 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.029325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.029618 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.029618 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.029659 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.029997 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.030002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.030354 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.031756 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.031757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.031757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.031906 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.032068 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.032069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.032069 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.032111 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.032333 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.032334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.032380 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.032449 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.032541 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:44.052537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39439
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:44.130465 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:44.130653 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:44.130676 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:44.131752 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:44.131796 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:44.131815 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:44.131837 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:44.133287 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:44.133458 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:44.133620 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:44.138137 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:44.138240 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:44.138286 1716 server_base.cc:1034] running on GCE node
W20260928 23:11:44.138127 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:44.138506 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:44.138688 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:44.139851 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790637104139834 us; error 31 us; skew 500 ppm
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:44.142119 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:44.142504 1716 webserver.cc:460] Webserver started at http://127.0.98.2:36597/ using document root <none> and password file <none>
I20260928 23:11:44.142668 1716 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:44.142704 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:44.142798 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:44.143723 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "13a4c09676444a08a5ebb41f125d28b4"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "8e42ac1b0405417352d39b4657a86cae"
server_key_iv: "db5c921ab55694585b5b9414ed4cb6fc"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.144017 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "13a4c09676444a08a5ebb41f125d28b4"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "8e42ac1b0405417352d39b4657a86cae"
server_key_iv: "db5c921ab55694585b5b9414ed4cb6fc"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.145099 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:11:44.145727 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.145893 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.145970 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "13a4c09676444a08a5ebb41f125d28b4"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "8e42ac1b0405417352d39b4657a86cae"
server_key_iv: "db5c921ab55694585b5b9414ed4cb6fc"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.146026 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:44.158219 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:44.159781 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:44.159914 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:44.160375 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:44.160413 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.160441 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:44.160461 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:44.167097 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42227
I20260928 23:11:44.167227 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42227 every 8 connection(s)
I20260928 23:11:44.167474 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:11:44.169294 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.160750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46427 (local address 127.0.98.62:39439)
0928 23:11:44.160808 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.160810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.161029 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:11:44.161684 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.161685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.161697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.161748 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.162546 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.163039 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.164005 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.164287 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.164351 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.166686 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.166688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.166688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.166695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.168101 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.168446 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.168448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.168448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.168505 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.168709 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.168709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.168710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.168816 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.168869 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.169113 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.169188 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:11:44.169448 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.160728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.2:46427)
0928 23:11:44.160897 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.160943 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.161586 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.161840 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.161844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.162174 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.162469 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.162476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.163121 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.163122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.163926 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.163928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.164395 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.164396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.164544 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.165038 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.165047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.166603 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.168167 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.168173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.168175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.168375 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.168556 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.168557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.168557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.168657 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.168866 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.168872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.168999 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.169083 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.169247 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 23:11:44.169854 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39439
I20260928 23:11:44.169965 1847 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:44.170157 1847 heartbeater.cc:511] Master 127.0.98.62:39439 requested a full tablet report, sending...
I20260928 23:11:44.170521 1503 ts_manager.cc:194] Registered new tserver with Master: 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2:42227)
I20260928 23:11:44.171080 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46427
I20260928 23:11:44.175710 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.171016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36053 (local address 127.0.98.62:39439)
0928 23:11:44.171063 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.171064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.171073 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:44.171096 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.171097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.171105 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.171153 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.171528 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.172044 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.172706 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.172918 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.172969 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.173817 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.173819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.173819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.173825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.174748 (+ 923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.175038 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.175039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.175040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.175090 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.175244 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.175244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.175245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.175374 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.175430 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.175560 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.175623 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:11:44.175724 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.170864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.2:36053)
0928 23:11:44.170967 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.170977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.171055 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.171204 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.171205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.171394 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.171460 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.171462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.172124 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.172125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.172642 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.172643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.172974 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.172975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.173035 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.173431 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.173435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.173694 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.174828 (+ 1134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.174829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.174829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.174981 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.175141 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.175142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.175142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.175196 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.175430 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.175431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.175472 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.175533 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.175624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260928 23:11:44.176541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1716
I20260928 23:11:44.176621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:44.176771 392 external_mini_cluster.cc:1428] Setting key a46886312e2f6b5978f9b16c7d824684
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:44.197521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39439
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:44.274338 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:44.274528 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:44.274554 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:44.275643 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:44.275677 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:44.275695 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:44.275710 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:44.277177 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:44.277379 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:44.277554 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:44.282107 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:44.282200 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:44.282150 1856 server_base.cc:1034] running on GCE node
W20260928 23:11:44.282107 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:44.282506 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:44.282714 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:44.283857 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790637104283843 us; error 28 us; skew 500 ppm
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:44.286074 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:44.286460 1856 webserver.cc:460] Webserver started at http://127.0.98.3:39247/ using document root <none> and password file <none>
I20260928 23:11:44.286650 1856 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:44.286695 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:44.286842 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:44.287822 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "4433b2c36d6047ff8c3cb67771de433c"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "6e6c832a46494172754ab899a1793447"
server_key_iv: "ef578e19c15a292846d6c279dcd8965c"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.288146 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4433b2c36d6047ff8c3cb67771de433c"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "6e6c832a46494172754ab899a1793447"
server_key_iv: "ef578e19c15a292846d6c279dcd8965c"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.289309 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:44.289922 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.290069 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:44.290151 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "4433b2c36d6047ff8c3cb67771de433c"
format_stamp: "Formatted at 2026-09-28 23:11:44 on dist-test-slave-6vgd"
server_key: "6e6c832a46494172754ab899a1793447"
server_key_iv: "ef578e19c15a292846d6c279dcd8965c"
server_key_version: "encryptionkey@0"
I20260928 23:11:44.290213 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:44.315671 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:44.317415 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:44.317548 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:44.318020 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:44.318060 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.318089 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:44.318114 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.324219 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44105
I20260928 23:11:44.324358 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44105 every 8 connection(s)
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:44.324595 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:11:44.327483 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.318482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36329 (local address 127.0.98.62:39439)
0928 23:11:44.318525 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.318527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.318712 (+ 185us) server_negotiation.cc:407] Connection header received
0928 23:11:44.319197 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.319199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.319212 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.319288 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.320071 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.320805 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.322290 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.322570 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.322644 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.324993 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.324994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.324995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.325002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.326286 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.326631 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.326632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.326632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.326686 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.326904 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.326905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.326905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.327014 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.327069 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.327311 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.327387 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:11:44.327629 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.318474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.3:36329)
0928 23:11:44.318608 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.318640 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.319109 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.319363 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.319366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.319693 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.319971 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.319978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.320898 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.320899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.322182 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.322185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.322649 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.322649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.322746 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.323253 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.323264 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.324896 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.326358 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.326364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.326366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.326568 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.326736 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.326737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.326737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.326850 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.327066 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.327072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.327199 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.327283 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.327428 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260928 23:11:44.328053 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39439
I20260928 23:11:44.328127 1987 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:44.328282 1987 heartbeater.cc:511] Master 127.0.98.62:39439 requested a full tablet report, sending...
I20260928 23:11:44.328634 1503 ts_manager.cc:194] Registered new tserver with Master: 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105)
I20260928 23:11:44.329022 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36329
I20260928 23:11:44.331489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 1856
I20260928 23:11:44.331583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:44.331751 392 external_mini_cluster.cc:1428] Setting key 4446a9006c636b585f6092b38b531e6d
I20260928 23:11:44.332543 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:11:44.334018 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.328952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.98.3:48561)
0928 23:11:44.329074 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.329084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.329142 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.329266 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.329267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.329455 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.329523 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.329525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.330116 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.330117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.330602 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.330604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.330886 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.330886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.330928 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.331278 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.331282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.331538 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.333155 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.333156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.333156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.333342 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.333518 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.333518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.333519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.333577 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.333807 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.333807 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.333837 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.333883 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.333950 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260928 23:11:44.334052 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.329023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48561 (local address 127.0.98.62:39439)
0928 23:11:44.329061 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.329062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.329130 (+ 68us) server_negotiation.cc:407] Connection header received
0928 23:11:44.329195 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.329196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.329205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.329263 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.329602 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.330044 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.330663 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.330842 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.330880 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.331615 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.331617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.331617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.331623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.333085 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.333404 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.333405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.333406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.333464 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.333627 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.333628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.333628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.333754 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.333813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.333911 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.333973 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:44.361092 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.354320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.0.98.62:39439)
0928 23:11:44.354382 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.354384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.354392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:44.354419 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.354420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.354429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.354496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.355216 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.355828 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.356590 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.356767 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.356809 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.358899 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.358907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.358907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.358915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.360142 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.360435 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.360436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.360437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.360489 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.360644 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.360645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.360645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.360750 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.360833 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.360932 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.361000 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:11:44.362607 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:44.364105 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:44.370533 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.0.1:49242)
0928 23:11:44.366476 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.366503 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.366555 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.367303 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.367304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.367306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.367345 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.367346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368477 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.368478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.369368 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.369370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370320 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370320 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.370370 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370373 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.370410 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.370439 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":55,"threads_started":1}
I20260928 23:11:44.370577 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.98.3:44105)
0928 23:11:44.367166 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.367170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.367178 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:44.367211 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.367213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.367227 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.367301 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.367408 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368422 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.369437 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370244 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370346 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.370447 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.370482 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":79,"threads_started":1}
I20260928 23:11:44.370533 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.0.1:34722)
0928 23:11:44.366454 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.366495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.366557 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.367052 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.367053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.367058 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.367139 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.367140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368095 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.368096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.369320 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.369323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370243 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.370304 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370317 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.370379 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.370439 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":16,"thread_start_us":49,"threads_started":1}
I20260928 23:11:44.370853 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42227 (local address 127.0.0.1:40434)
0928 23:11:44.366705 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.366712 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.366761 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.367049 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.367052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.367054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.367152 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.367155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368413 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.368414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.369475 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.369477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370487 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.370547 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.370625 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.370663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":21,"thread_start_us":66,"threads_started":1}
I20260928 23:11:44.371065 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.0.98.2:42227)
0928 23:11:44.366702 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.366708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.366754 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:11:44.366872 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.366875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.366893 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.366965 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.367230 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368335 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.369554 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370277 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370342 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370771 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.370941 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.370978 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260928 23:11:44.371225 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.366271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.0.98.1:36587)
0928 23:11:44.366917 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.366922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.366931 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:44.366965 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.366966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.366978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.367049 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.367212 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.368021 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.369408 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.370069 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.370149 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.370217 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.370575 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.371125 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":84,"threads_started":1}
I20260928 23:11:44.371305 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 56ed922adf144564a3ddbb8807bda97e (DEFAULT_TABLE table=test-table [id=89af997ea0f942248fd76303e34a2e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.371685 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ed922adf144564a3ddbb8807bda97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.371798 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 56ed922adf144564a3ddbb8807bda97e (DEFAULT_TABLE table=test-table [id=89af997ea0f942248fd76303e34a2e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.372097 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ed922adf144564a3ddbb8807bda97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.373639 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 56ed922adf144564a3ddbb8807bda97e (DEFAULT_TABLE table=test-table [id=89af997ea0f942248fd76303e34a2e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.373970 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ed922adf144564a3ddbb8807bda97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.374763 2007 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Bootstrap starting.
I20260928 23:11:44.375017 2008 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Bootstrap starting.
I20260928 23:11:44.375648 2008 tablet_bootstrap.cc:654] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.375902 2008 log.cc:824] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:44.376551 2007 tablet_bootstrap.cc:654] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.376595 2008 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: No bootstrap required, opened a new log
I20260928 23:11:44.376652 2008 ts_tablet_manager.cc:1397] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:44.376812 2007 log.cc:824] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:44.377256 2011 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Bootstrap starting.
I20260928 23:11:44.377622 2007 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: No bootstrap required, opened a new log
I20260928 23:11:44.377722 2007 ts_tablet_manager.cc:1397] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:11:44.377827 2011 tablet_bootstrap.cc:654] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.378121 2011 log.cc:824] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:44.378626 2008 raft_consensus.cc:348] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.378751 2008 raft_consensus.cc:374] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.378780 2008 raft_consensus.cc:729] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd6cbd97cb74e9cb9a5c3efe31deb97, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.378849 2011 tablet_bootstrap.cc:492] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: No bootstrap required, opened a new log
I20260928 23:11:44.378911 2011 ts_tablet_manager.cc:1397] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:44.378901 2008 consensus_queue.cc:260] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.379171 2008 ts_tablet_manager.cc:1428] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:11:44.379176 1707 heartbeater.cc:503] Master 127.0.98.62:39439 was elected leader, sending a full tablet report...
I20260928 23:11:44.379578 2007 raft_consensus.cc:348] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.379731 2007 raft_consensus.cc:374] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.379765 2007 raft_consensus.cc:729] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4433b2c36d6047ff8c3cb67771de433c, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.379889 2007 consensus_queue.cc:260] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.380149 2007 ts_tablet_manager.cc:1428] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:11:44.380252 1987 heartbeater.cc:503] Master 127.0.98.62:39439 was elected leader, sending a full tablet report...
I20260928 23:11:44.380738 2011 raft_consensus.cc:348] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.380880 2011 raft_consensus.cc:374] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.380903 2011 raft_consensus.cc:729] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a4c09676444a08a5ebb41f125d28b4, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.380977 2011 consensus_queue.cc:260] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.381150 2011 ts_tablet_manager.cc:1428] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:11:44.381197 1847 heartbeater.cc:503] Master 127.0.98.62:39439 was elected leader, sending a full tablet report...
W20260928 23:11:44.418022 1848 tablet.cc:2369] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:44.523051 2015 raft_consensus.cc:482] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:44.523195 2015 raft_consensus.cc:504] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.523582 2015 leader_election.cc:288] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105)
W20260928 23:11:44.524066 1708 tablet.cc:2369] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:44.526866 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.523848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60723 (local address 127.0.98.1:36587)
0928 23:11:44.524534 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.524536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.524542 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:11:44.524565 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.524566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.524578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.524620 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.524787 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.525259 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526175 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.526569 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526628 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.526638 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.526773 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.526807 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666}
I20260928 23:11:44.526891 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.523915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52683 (local address 127.0.98.3:44105)
0928 23:11:44.523960 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.523962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.524483 (+ 521us) server_negotiation.cc:407] Connection header received
0928 23:11:44.524515 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.524516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.524524 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.524575 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.524724 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.525188 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526153 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.526596 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526657 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.526666 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.526794 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.526826 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:44.526912 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.523769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.98.2:52683)
0928 23:11:44.524418 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.524428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.524475 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.524592 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.524593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.524594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.524667 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.524668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.525276 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.525277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.526055 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.526057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.526663 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.526736 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.526739 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.526783 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.526821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":66,"thread_start_us":96,"threads_started":1}
I20260928 23:11:44.526916 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.523769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.98.2:60723)
0928 23:11:44.523977 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.523988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.524064 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.524657 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.524657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.524659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.524707 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.524709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.525337 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.525338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.526067 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.526069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.526636 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.526637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.526714 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.526718 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.526762 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.526799 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260928 23:11:44.527292 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ed922adf144564a3ddbb8807bda97e" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4433b2c36d6047ff8c3cb67771de433c" is_pre_election: true
I20260928 23:11:44.527379 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ed922adf144564a3ddbb8807bda97e" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" is_pre_election: true
I20260928 23:11:44.527432 1941 raft_consensus.cc:2454] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 0.
I20260928 23:11:44.527511 1661 raft_consensus.cc:2454] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 0.
I20260928 23:11:44.527621 1733 leader_election.cc:302] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a4c09676444a08a5ebb41f125d28b4, 4433b2c36d6047ff8c3cb67771de433c; no voters:
I20260928 23:11:44.527737 2015 raft_consensus.cc:2781] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:44.527781 2015 raft_consensus.cc:482] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:44.527809 2015 raft_consensus.cc:3037] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.528725 2015 raft_consensus.cc:504] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.528833 2015 leader_election.cc:288] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105)
I20260928 23:11:44.528987 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ed922adf144564a3ddbb8807bda97e" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
I20260928 23:11:44.529059 1661 raft_consensus.cc:3037] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.529045 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ed922adf144564a3ddbb8807bda97e" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4433b2c36d6047ff8c3cb67771de433c"
I20260928 23:11:44.529124 1941 raft_consensus.cc:3037] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.529732 1661 raft_consensus.cc:2454] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 1.
I20260928 23:11:44.529793 1941 raft_consensus.cc:2454] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 1.
I20260928 23:11:44.529872 1735 leader_election.cc:302] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a4c09676444a08a5ebb41f125d28b4, 8fd6cbd97cb74e9cb9a5c3efe31deb97; no voters:
I20260928 23:11:44.529948 2015 raft_consensus.cc:2781] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:44.530067 2015 raft_consensus.cc:686] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 1 LEADER]: Becoming Leader. State: Replica: 13a4c09676444a08a5ebb41f125d28b4, State: Running, Role: LEADER
I20260928 23:11:44.530148 2015 consensus_queue.cc:237] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } }
I20260928 23:11:44.530952 1503 catalog_manager.cc:5495] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "13a4c09676444a08a5ebb41f125d28b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } health_report { overall_health: HEALTHY } } }
I20260928 23:11:44.555774 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.553503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.0.98.2:42227)
0928 23:11:44.553547 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.553549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.553572 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:11:44.553673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.553674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.553690 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.553741 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:44.553871 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.554326 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.555079 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.555255 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.555306 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.555498 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.555586 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.555650 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.555688 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.555707 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:44.557379 1941 raft_consensus.cc:1261] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Refusing update from remote peer 13a4c09676444a08a5ebb41f125d28b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.557379 1661 raft_consensus.cc:1261] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Refusing update from remote peer 13a4c09676444a08a5ebb41f125d28b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.557679 2017 consensus_queue.cc:1035] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.557796 2015 consensus_queue.cc:1035] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.560575 2021 mvcc.cc:205] Tried to move back new op lower bound from 7334449580264378368 to 7334449580155838464. Current Snapshot: MvccSnapshot[applied={T|T < 7334449580264378368}]
I20260928 23:11:44.563031 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:44.563105 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:44.563769 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Sending DeleteTablet for 3 replicas of tablet 56ed922adf144564a3ddbb8807bda97e
I20260928 23:11:44.564090 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ed922adf144564a3ddbb8807bda97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49242
I20260928 23:11:44.564087 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ed922adf144564a3ddbb8807bda97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34722
I20260928 23:11:44.564349 2030 tablet_replica.cc:330] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: stopping tablet replica
I20260928 23:11:44.564349 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ed922adf144564a3ddbb8807bda97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260928 23:11:44.564445 2030 raft_consensus.cc:2229] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:44.564549 2030 raft_consensus.cc:2258] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.564491 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:44.564612 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:44.564680 2029 tablet_replica.cc:330] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: stopping tablet replica
I20260928 23:11:44.564891 2029 raft_consensus.cc:2229] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:44.565003 2029 raft_consensus.cc:2258] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.565315 2030 ts_tablet_manager.cc:1905] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.565831 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 (DEFAULT_TABLE table=test-table [id=92f91cabb47f4464a6861e6fe5debbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.565855 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 (DEFAULT_TABLE table=test-table [id=92f91cabb47f4464a6861e6fe5debbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.565944 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861f8799b3b477bbbd75d193f6c1bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.565964 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861f8799b3b477bbbd75d193f6c1bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.565981 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 (DEFAULT_TABLE table=test-table [id=92f91cabb47f4464a6861e6fe5debbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:44.566056 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1861f8799b3b477bbbd75d193f6c1bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.567183 2011 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Bootstrap starting.
I20260928 23:11:44.567183 2007 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Bootstrap starting.
I20260928 23:11:44.567209 2008 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Bootstrap starting.
I20260928 23:11:44.567552 2008 tablet_bootstrap.cc:654] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.567561 2007 tablet_bootstrap.cc:654] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.567561 2011 tablet_bootstrap.cc:654] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.568137 2008 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: No bootstrap required, opened a new log
I20260928 23:11:44.568188 2008 ts_tablet_manager.cc:1397] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:44.568337 2008 raft_consensus.cc:348] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.568405 2008 raft_consensus.cc:374] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.568429 2008 raft_consensus.cc:729] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd6cbd97cb74e9cb9a5c3efe31deb97, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.568444 2011 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: No bootstrap required, opened a new log
I20260928 23:11:44.568465 2008 consensus_queue.cc:260] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.568485 2011 ts_tablet_manager.cc:1397] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:44.568565 2008 ts_tablet_manager.cc:1428] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:44.568601 2011 raft_consensus.cc:348] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.568684 2011 raft_consensus.cc:374] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.568712 2011 raft_consensus.cc:729] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a4c09676444a08a5ebb41f125d28b4, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.568748 2011 consensus_queue.cc:260] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.568837 2011 ts_tablet_manager.cc:1428] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:44.568915 2007 tablet_bootstrap.cc:492] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: No bootstrap required, opened a new log
I20260928 23:11:44.568957 2007 ts_tablet_manager.cc:1397] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:11:44.569093 2007 raft_consensus.cc:348] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.569183 2007 raft_consensus.cc:374] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.569203 2007 raft_consensus.cc:729] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4433b2c36d6047ff8c3cb67771de433c, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.569259 2007 consensus_queue.cc:260] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.569336 2007 ts_tablet_manager.cc:1428] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:44.570211 2030 ts_tablet_manager.cc:1918] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:11:44.570276 2030 log.cc:1196] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/56ed922adf144564a3ddbb8807bda97e
I20260928 23:11:44.570442 2029 ts_tablet_manager.cc:1905] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.570498 2030 ts_tablet_manager.cc:1939] T 56ed922adf144564a3ddbb8807bda97e P 4433b2c36d6047ff8c3cb67771de433c: Deleting consensus metadata
I20260928 23:11:44.570115 2031 tablet_replica.cc:330] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: stopping tablet replica
I20260928 23:11:44.570885 2031 raft_consensus.cc:2229] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:11:44.570899 1489 catalog_manager.cc:4841] TS 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105): tablet 56ed922adf144564a3ddbb8807bda97e (table test-table [id=89af997ea0f942248fd76303e34a2e37]) successfully deleted
I20260928 23:11:44.570986 2031 raft_consensus.cc:2258] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.571360 2031 ts_tablet_manager.cc:1905] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.572458 2029 ts_tablet_manager.cc:1918] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:11:44.572530 2029 log.cc:1196] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/56ed922adf144564a3ddbb8807bda97e
I20260928 23:11:44.572752 2029 ts_tablet_manager.cc:1939] T 56ed922adf144564a3ddbb8807bda97e P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting consensus metadata
I20260928 23:11:44.573114 1491 catalog_manager.cc:4841] TS 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587): tablet 56ed922adf144564a3ddbb8807bda97e (table test-table [id=89af997ea0f942248fd76303e34a2e37]) successfully deleted
I20260928 23:11:44.573237 2031 ts_tablet_manager.cc:1918] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:11:44.573298 2031 log.cc:1196] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/56ed922adf144564a3ddbb8807bda97e
I20260928 23:11:44.573487 2031 ts_tablet_manager.cc:1939] T 56ed922adf144564a3ddbb8807bda97e P 13a4c09676444a08a5ebb41f125d28b4: Deleting consensus metadata
I20260928 23:11:44.573800 1491 catalog_manager.cc:4841] TS 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2:42227): tablet 56ed922adf144564a3ddbb8807bda97e (table test-table [id=89af997ea0f942248fd76303e34a2e37]) successfully deleted
W20260928 23:11:44.575095 1988 tablet.cc:2369] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:44.610136 2015 raft_consensus.cc:482] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:44.610219 2015 raft_consensus.cc:504] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.610368 2015 leader_election.cc:288] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105)
I20260928 23:11:44.610525 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861f8799b3b477bbbd75d193f6c1bb5" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" is_pre_election: true
I20260928 23:11:44.610553 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861f8799b3b477bbbd75d193f6c1bb5" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4433b2c36d6047ff8c3cb67771de433c" is_pre_election: true
I20260928 23:11:44.610622 1941 raft_consensus.cc:2454] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 0.
I20260928 23:11:44.610620 1661 raft_consensus.cc:2454] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 0.
I20260928 23:11:44.610749 1735 leader_election.cc:302] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a4c09676444a08a5ebb41f125d28b4, 8fd6cbd97cb74e9cb9a5c3efe31deb97; no voters:
I20260928 23:11:44.610918 2015 raft_consensus.cc:2781] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:44.610993 2015 raft_consensus.cc:482] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:44.611017 2015 raft_consensus.cc:3037] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.611641 2015 raft_consensus.cc:504] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.611747 2015 leader_election.cc:288] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 election: Requested vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105)
I20260928 23:11:44.611904 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861f8799b3b477bbbd75d193f6c1bb5" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
I20260928 23:11:44.611939 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1861f8799b3b477bbbd75d193f6c1bb5" candidate_uuid: "13a4c09676444a08a5ebb41f125d28b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4433b2c36d6047ff8c3cb67771de433c"
I20260928 23:11:44.611985 1661 raft_consensus.cc:3037] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.612010 1941 raft_consensus.cc:3037] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.612555 1941 raft_consensus.cc:2454] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 1.
I20260928 23:11:44.612687 1733 leader_election.cc:302] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a4c09676444a08a5ebb41f125d28b4, 4433b2c36d6047ff8c3cb67771de433c; no voters:
I20260928 23:11:44.612725 1661 raft_consensus.cc:2454] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13a4c09676444a08a5ebb41f125d28b4 in term 1.
I20260928 23:11:44.612762 2015 raft_consensus.cc:2781] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:44.612802 2015 raft_consensus.cc:686] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 LEADER]: Becoming Leader. State: Replica: 13a4c09676444a08a5ebb41f125d28b4, State: Running, Role: LEADER
I20260928 23:11:44.612847 2015 consensus_queue.cc:237] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.613327 1503 catalog_manager.cc:5495] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "13a4c09676444a08a5ebb41f125d28b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:44.648850 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.641901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39439 (local address 127.0.0.1:46132)
0928 23:11:44.642039 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.642048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.642113 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.642238 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.642238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.642537 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:44.642582 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.642584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.643230 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.643231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.643749 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.643751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.644096 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.644097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.644152 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.644612 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:44.644616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:44.646167 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:44.647844 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.647851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.647853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.648087 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.648299 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:44.648300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:44.648300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:44.648354 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:44.648585 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:44.648591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:44.648635 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.648687 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.648773 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 23:11:44.648943 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.641947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.98.62:39439)
0928 23:11:44.641996 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.641998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.642104 (+ 106us) server_negotiation.cc:407] Connection header received
0928 23:11:44.642171 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.642172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.642182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.642230 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.642661 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.643163 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.643817 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.644037 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.644097 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.646246 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.646248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.646248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.646255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.647750 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.648171 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.648172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.648173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.648238 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.648402 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.648403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.648403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.648530 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.648585 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.648722 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.648834 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":25}
I20260928 23:11:44.650895 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46132:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:11:44.651113 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:44.652348 1921 tablet_service.cc:1467] Processing CreateTablet for tablet dc874898a7534d4aaaac559a2749d168 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8aa44f062a2b45bd8851d1cfe27a3aff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:44.652431 1641 tablet_service.cc:1467] Processing CreateTablet for tablet dc874898a7534d4aaaac559a2749d168 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8aa44f062a2b45bd8851d1cfe27a3aff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:44.652524 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc874898a7534d4aaaac559a2749d168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.652585 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc874898a7534d4aaaac559a2749d168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.653637 2007 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: Bootstrap starting.
I20260928 23:11:44.653702 1781 tablet_service.cc:1467] Processing CreateTablet for tablet dc874898a7534d4aaaac559a2749d168 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8aa44f062a2b45bd8851d1cfe27a3aff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:44.653821 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc874898a7534d4aaaac559a2749d168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:44.654032 2007 tablet_bootstrap.cc:654] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.654335 2008 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Bootstrap starting.
I20260928 23:11:44.654685 2008 tablet_bootstrap.cc:654] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.655171 2011 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: Bootstrap starting.
I20260928 23:11:44.655318 2008 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: No bootstrap required, opened a new log
I20260928 23:11:44.655368 2008 ts_tablet_manager.cc:1397] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:44.655503 2008 raft_consensus.cc:348] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.655552 2011 tablet_bootstrap.cc:654] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:44.655577 2008 raft_consensus.cc:374] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.655597 2008 raft_consensus.cc:729] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fd6cbd97cb74e9cb9a5c3efe31deb97, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.655643 2008 consensus_queue.cc:260] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.655728 2008 ts_tablet_manager.cc:1428] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:44.655875 2013 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.655963 2013 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.656307 2007 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: No bootstrap required, opened a new log
I20260928 23:11:44.656316 2011 tablet_bootstrap.cc:492] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: No bootstrap required, opened a new log
I20260928 23:11:44.656353 2011 ts_tablet_manager.cc:1397] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:44.656353 2007 ts_tablet_manager.cc:1397] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 23:11:44.656486 2007 raft_consensus.cc:348] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.656486 2011 raft_consensus.cc:348] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.656551 2007 raft_consensus.cc:374] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.656551 2011 raft_consensus.cc:374] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:44.656570 2011 raft_consensus.cc:729] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a4c09676444a08a5ebb41f125d28b4, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.656570 2007 raft_consensus.cc:729] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4433b2c36d6047ff8c3cb67771de433c, State: Initialized, Role: FOLLOWER
I20260928 23:11:44.656612 2011 consensus_queue.cc:260] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.656611 2007 consensus_queue.cc:260] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.656687 2007 ts_tablet_manager.cc:1428] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.656685 2011 ts_tablet_manager.cc:1428] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:44.656760 2015 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.656826 2015 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.656955 2014 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.657037 2014 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.839776 2014 raft_consensus.cc:482] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:44.839860 2014 raft_consensus.cc:504] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.840128 2014 leader_election.cc:288] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2:42227)
I20260928 23:11:44.842924 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.840344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38387 (local address 127.0.98.1:36587)
0928 23:11:44.840401 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.840402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.840461 (+ 59us) server_negotiation.cc:407] Connection header received
0928 23:11:44.840559 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.840561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.840571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.840601 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.840799 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.841314 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842279 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.842625 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842683 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.842694 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.842815 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.842849 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:11:44.842926 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.840500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39669 (local address 127.0.98.2:42227)
0928 23:11:44.840563 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.840565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.840574 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:44.840622 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.840624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.840631 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.840684 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:44.840804 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.841311 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842276 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.842643 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842702 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.842711 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.842821 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.842855 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:11:44.842948 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.840432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42227 (local address 127.0.98.3:39669)
0928 23:11:44.840509 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.840520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.840578 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.840688 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.840688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.840690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.840746 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.840747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.841377 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.841378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.842198 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.842200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.842708 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.842766 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.842769 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.842811 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.842850 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":40,"thread_start_us":36,"threads_started":1}
I20260928 23:11:44.842947 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.840280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36587 (local address 127.0.98.3:38387)
0928 23:11:44.840386 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.840396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.840469 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.840668 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.840669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.840671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:44.840739 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.840740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.841381 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.841382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.842196 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.842198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.842695 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.842696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.842758 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.842761 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.842804 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.842845 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260928 23:11:44.843259 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc874898a7534d4aaaac559a2749d168" candidate_uuid: "4433b2c36d6047ff8c3cb67771de433c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a4c09676444a08a5ebb41f125d28b4" is_pre_election: true
I20260928 23:11:44.843398 1801 raft_consensus.cc:2454] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4433b2c36d6047ff8c3cb67771de433c in term 0.
I20260928 23:11:44.843447 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc874898a7534d4aaaac559a2749d168" candidate_uuid: "4433b2c36d6047ff8c3cb67771de433c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" is_pre_election: true
I20260928 23:11:44.843537 1661 raft_consensus.cc:2454] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4433b2c36d6047ff8c3cb67771de433c in term 0.
I20260928 23:11:44.843590 1875 leader_election.cc:302] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13a4c09676444a08a5ebb41f125d28b4, 4433b2c36d6047ff8c3cb67771de433c; no voters:
I20260928 23:11:44.843715 2014 raft_consensus.cc:2781] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:44.843757 2014 raft_consensus.cc:482] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:44.843776 2014 raft_consensus.cc:3037] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.844420 2014 raft_consensus.cc:504] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.844528 2014 leader_election.cc:288] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [CANDIDATE]: Term 1 election: Requested vote from peers 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587), 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2:42227)
I20260928 23:11:44.844694 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc874898a7534d4aaaac559a2749d168" candidate_uuid: "4433b2c36d6047ff8c3cb67771de433c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13a4c09676444a08a5ebb41f125d28b4"
I20260928 23:11:44.844686 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc874898a7534d4aaaac559a2749d168" candidate_uuid: "4433b2c36d6047ff8c3cb67771de433c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97"
I20260928 23:11:44.844762 1661 raft_consensus.cc:3037] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.844761 1801 raft_consensus.cc:3037] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:44.845517 1661 raft_consensus.cc:2454] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4433b2c36d6047ff8c3cb67771de433c in term 1.
I20260928 23:11:44.845587 1801 raft_consensus.cc:2454] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4433b2c36d6047ff8c3cb67771de433c in term 1.
I20260928 23:11:44.845679 1875 leader_election.cc:302] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4433b2c36d6047ff8c3cb67771de433c, 8fd6cbd97cb74e9cb9a5c3efe31deb97; no voters:
I20260928 23:11:44.845762 2014 raft_consensus.cc:2781] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:44.845881 2014 raft_consensus.cc:686] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [term 1 LEADER]: Becoming Leader. State: Replica: 4433b2c36d6047ff8c3cb67771de433c, State: Running, Role: LEADER
I20260928 23:11:44.845942 2014 consensus_queue.cc:237] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } }
I20260928 23:11:44.846231 2047 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: TxnStatusTablet state changed. Reason: New leader 4433b2c36d6047ff8c3cb67771de433c. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.846343 2047 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:44.846526 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:11:44.846627 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:11:44.846607 1504 catalog_manager.cc:5495] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } health_report { overall_health: HEALTHY } } }
I20260928 23:11:44.880574 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.878300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.0.1:49248)
0928 23:11:44.878359 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.878369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.878407 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.878579 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.878580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.878581 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:44.878627 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.878630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.879242 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.879243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.879799 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.879800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.880122 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.880123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.880176 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.880189 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.880409 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.880416 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.880458 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.880509 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:11:44.880576 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.878382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.3:44105)
0928 23:11:44.878439 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.878441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.878449 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:44.878481 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.878482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.878514 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.878571 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:44.878681 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.879180 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.879867 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.880065 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.880116 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.880264 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.880358 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.880416 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.880460 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.880481 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:11:44.881664 1661 raft_consensus.cc:1261] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Refusing update from remote peer 4433b2c36d6047ff8c3cb67771de433c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.881762 1801 raft_consensus.cc:1261] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Refusing update from remote peer 4433b2c36d6047ff8c3cb67771de433c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.882047 2014 consensus_queue.cc:1035] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.882180 2047 consensus_queue.cc:1035] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c [LEADER]: Connected to new peer: Peer: permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.882733 2013 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: TxnStatusTablet state changed. Reason: New leader 4433b2c36d6047ff8c3cb67771de433c. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.882830 2013 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.882783 2015 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: TxnStatusTablet state changed. Reason: New leader 4433b2c36d6047ff8c3cb67771de433c. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.882861 2015 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.883658 2047 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.883734 2047 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:44.883697 2014 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.883764 2014 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 4433b2c36d6047ff8c3cb67771de433c: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:44.884191 2013 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.884274 2013 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.884662 2015 tablet_replica.cc:439] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4433b2c36d6047ff8c3cb67771de433c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 } } peers { permanent_uuid: "13a4c09676444a08a5ebb41f125d28b4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42227 } } peers { permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 } } }
I20260928 23:11:44.884739 2015 tablet_replica.cc:442] T dc874898a7534d4aaaac559a2749d168 P 13a4c09676444a08a5ebb41f125d28b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:44.889665 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.887223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39053 (local address 127.0.98.3:44105)
0928 23:11:44.887276 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.887278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.887373 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:11:44.887441 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.887442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.887449 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.887492 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:44.887596 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.888023 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.889091 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.889310 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.889353 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.889508 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.889529 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.889557 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.889609 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.889620 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:11:44.889729 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.887146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44105 (local address 127.0.98.2:39053)
0928 23:11:44.887317 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.887328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.887388 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.887495 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.887496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.887498 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:44.887546 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.887548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.888079 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.888080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.889019 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.889022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.889363 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.889364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.889423 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.889437 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.889560 (+ 123us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.889562 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.889607 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.889651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260928 23:11:44.891397 1941 raft_consensus.cc:1261] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Refusing update from remote peer 13a4c09676444a08a5ebb41f125d28b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.891532 1661 raft_consensus.cc:1261] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Refusing update from remote peer 13a4c09676444a08a5ebb41f125d28b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:44.891593 2015 consensus_queue.cc:1035] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4433b2c36d6047ff8c3cb67771de433c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.891788 2015 consensus_queue.cc:1035] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fd6cbd97cb74e9cb9a5c3efe31deb97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:44.893015 2020 mvcc.cc:205] Tried to move back new op lower bound from 7334449581633646592 to 7334449580494241792. Current Snapshot: MvccSnapshot[applied={T|T < 7334449581633646592}]
I20260928 23:11:44.893741 2022 mvcc.cc:205] Tried to move back new op lower bound from 7334449581633646592 to 7334449580494241792. Current Snapshot: MvccSnapshot[applied={T|T < 7334449581633646592}]
I20260928 23:11:44.899859 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.897279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44731 (local address 127.0.98.2:42227)
0928 23:11:44.897313 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.897315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.897511 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:11:44.897534 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.897535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.897545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.897606 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:44.897682 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.898395 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.899239 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.899466 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.899523 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.899658 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.899690 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.899730 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.899780 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.899798 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:11:44.899892 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.897231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42227 (local address 127.0.98.3:44731)
0928 23:11:44.897434 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:44.897444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:44.897502 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:44.897606 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:44.897607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:44.897608 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:44.897640 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.897641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.898459 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.898459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.899147 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:44.899150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.899530 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.899531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:44.899583 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.899596 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:44.899731 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:44.899732 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:11:44.899777 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:11:44.899815 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
I20260928 23:11:44.908046 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:44.908128 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:44.908684 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87854f80d492429587659a6745f3ea3b: Sending DeleteTablet for 3 replicas of tablet 1861f8799b3b477bbbd75d193f6c1bb5
I20260928 23:11:44.908914 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49242
I20260928 23:11:44.908944 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40434
I20260928 23:11:44.908962 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 1861f8799b3b477bbbd75d193f6c1bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34722
I20260928 23:11:44.909008 2030 tablet_replica.cc:330] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: stopping tablet replica
I20260928 23:11:44.909060 2030 raft_consensus.cc:2229] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:44.909053 2029 tablet_replica.cc:330] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: stopping tablet replica
I20260928 23:11:44.909091 2030 raft_consensus.cc:2258] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.909101 2029 raft_consensus.cc:2229] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:44.909129 2029 raft_consensus.cc:2258] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.909152 2031 tablet_replica.cc:330] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: stopping tablet replica
I20260928 23:11:44.909196 2031 raft_consensus.cc:2229] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:11:44.909260 2031 raft_consensus.cc:2258] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:44.909430 2029 ts_tablet_manager.cc:1905] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.909538 2030 ts_tablet_manager.cc:1905] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.910959 2031 ts_tablet_manager.cc:1905] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:44.911582 2029 ts_tablet_manager.cc:1918] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:44.911641 2029 log.cc:1196] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/1861f8799b3b477bbbd75d193f6c1bb5
I20260928 23:11:44.911830 2029 ts_tablet_manager.cc:1939] T 1861f8799b3b477bbbd75d193f6c1bb5 P 8fd6cbd97cb74e9cb9a5c3efe31deb97: Deleting consensus metadata
I20260928 23:11:44.912117 1491 catalog_manager.cc:4841] TS 8fd6cbd97cb74e9cb9a5c3efe31deb97 (127.0.98.1:36587): tablet 1861f8799b3b477bbbd75d193f6c1bb5 (table test-table [id=92f91cabb47f4464a6861e6fe5debbbd]) successfully deleted
I20260928 23:11:44.912366 2031 ts_tablet_manager.cc:1918] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:44.912410 2031 log.cc:1196] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/1861f8799b3b477bbbd75d193f6c1bb5
I20260928 23:11:44.912569 2031 ts_tablet_manager.cc:1939] T 1861f8799b3b477bbbd75d193f6c1bb5 P 13a4c09676444a08a5ebb41f125d28b4: Deleting consensus metadata
I20260928 23:11:44.912734 2030 ts_tablet_manager.cc:1918] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:44.912779 2030 log.cc:1196] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/1861f8799b3b477bbbd75d193f6c1bb5
I20260928 23:11:44.912917 1491 catalog_manager.cc:4841] TS 13a4c09676444a08a5ebb41f125d28b4 (127.0.98.2:42227): tablet 1861f8799b3b477bbbd75d193f6c1bb5 (table test-table [id=92f91cabb47f4464a6861e6fe5debbbd]) successfully deleted
I20260928 23:11:44.912950 2030 ts_tablet_manager.cc:1939] T 1861f8799b3b477bbbd75d193f6c1bb5 P 4433b2c36d6047ff8c3cb67771de433c: Deleting consensus metadata
I20260928 23:11:44.913247 1489 catalog_manager.cc:4841] TS 4433b2c36d6047ff8c3cb67771de433c (127.0.98.3:44105): tablet 1861f8799b3b477bbbd75d193f6c1bb5 (table test-table [id=92f91cabb47f4464a6861e6fe5debbbd]) successfully deleted
Sep 28 23:11:44 dist-test-slave-6vgd krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:11:44.920861 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.911708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.0.98.1:36587)
0928 23:11:44.912673 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.912675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.913250 (+ 575us) server_negotiation.cc:407] Connection header received
0928 23:11:44.913310 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.913311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.913323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.913374 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.913868 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.914585 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.915259 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.915528 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.915600 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.918382 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.918388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.918388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.918394 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.919805 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.920186 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.920188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.920188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.920236 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.920410 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.920411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.920411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.920535 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.920638 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.920736 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.920795 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942}
W20260928 23:11:44.921254 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34728
I20260928 23:11:44.928197 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:44.922144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.98.62:39439)
0928 23:11:44.922188 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:44.922190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:44.922239 (+ 49us) server_negotiation.cc:407] Connection header received
0928 23:11:44.922285 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:44.922286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:44.922296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:44.922363 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:44.922801 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.923538 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.924235 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:44.924514 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:44.924588 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:44.925679 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:44.925681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:44.925681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:44.925688 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:44.927140 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.927426 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.927427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.927428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.927489 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:44.927676 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:44.927677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:44.927677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:44.927822 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:44.927932 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:44.927998 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:11:44.928109 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 23:11:44.928428 1503 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46138
I20260928 23:11:44.929179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1576
I20260928 23:11:44.935536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1716
I20260928 23:11:44.941205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1856
I20260928 23:11:44.945547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 1472
2026-09-28T23:11:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2259 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:45 dist-test-slave-6vgd krb5kdc[2071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:45 dist-test-slave-6vgd krb5kdc[2071](info): set up 2 sockets
Sep 28 23:11:45 dist-test-slave-6vgd krb5kdc[2071](info): commencing operation
krb5kdc: starting...
W20260928 23:11:45.888717 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:45 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:45.995467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:46.072399 2084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:46.072608 2084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:46.072629 2084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:46.073727 2084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:46.073761 2084 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:11:46.073774 2084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:46.073786 2084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:46.073801 2084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:46.073812 2084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:46.075212 2084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37417
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:46.075399 2084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:46.075567 2084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:46.080060 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.080202 2084 server_base.cc:1034] running on GCE node
W20260928 23:11:46.080149 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:46.080051 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.080541 2084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:46.080827 2084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:46.081964 2084 hybrid_clock.cc:648] HybridClock initialized: now 1790637106081945 us; error 35 us; skew 500 ppm
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:46.084267 2084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:46.084666 2084 webserver.cc:460] Webserver started at http://127.0.98.62:38109/ using document root <none> and password file <none>
I20260928 23:11:46.084870 2084 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:46.084918 2084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:46.085037 2084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:46.086097 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "c94a067c63fd45c2a1b11f8d80c33667"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "f49bc61a8c6965209d2ae8a98d44562e"
server_key_iv: "334a5961b33ea776252810187620e694"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.086436 2084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "c94a067c63fd45c2a1b11f8d80c33667"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "f49bc61a8c6965209d2ae8a98d44562e"
server_key_iv: "334a5961b33ea776252810187620e694"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.087632 2084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:11:46.088231 2099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.088382 2084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:46.088461 2084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "c94a067c63fd45c2a1b11f8d80c33667"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "f49bc61a8c6965209d2ae8a98d44562e"
server_key_iv: "334a5961b33ea776252810187620e694"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.088526 2084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:46.132794 2084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:46.134792 2084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:46.134925 2084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:46.139237 2084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37417
I20260928 23:11:46.139285 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37417 every 8 connection(s)
I20260928 23:11:46.139603 2084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:46.139756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2084
I20260928 23:11:46.139840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:46.140017 392 external_mini_cluster.cc:1428] Setting key deb1ec30a6434f0ab700c283a76e7c04
I20260928 23:11:46.140288 2162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.143261 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Bootstrap starting.
I20260928 23:11:46.143929 2162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.144273 2162 log.cc:824] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:46.144882 2162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: No bootstrap required, opened a new log
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.147189 2162 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } }
I20260928 23:11:46.147326 2162 raft_consensus.cc:374] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.147348 2162 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94a067c63fd45c2a1b11f8d80c33667, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.147469 2162 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } }
I20260928 23:11:46.147533 2162 raft_consensus.cc:388] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:46.147562 2162 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:46.147609 2162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:46.148082 2162 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } }
I20260928 23:11:46.148176 2162 leader_election.cc:302] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c94a067c63fd45c2a1b11f8d80c33667; no voters:
I20260928 23:11:46.148320 2162 leader_election.cc:288] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:46.148367 2167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:46.148537 2167 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [term 1 LEADER]: Becoming Leader. State: Replica: c94a067c63fd45c2a1b11f8d80c33667, State: Running, Role: LEADER
I20260928 23:11:46.148553 2162 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:46.148708 2167 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } }
I20260928 23:11:46.149295 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c94a067c63fd45c2a1b11f8d80c33667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } } }
I20260928 23:11:46.149442 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:46.149698 2177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:46.150187 2177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:46.150303 2168 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94a067c63fd45c2a1b11f8d80c33667. Latest consensus state: current_term: 1 leader_uuid: "c94a067c63fd45c2a1b11f8d80c33667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94a067c63fd45c2a1b11f8d80c33667" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37417 } } }
I20260928 23:11:46.150480 2168 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:46.150825 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.140518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.0.98.62:37417)
0928 23:11:46.140673 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.140682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.140709 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:11:46.141388 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.141396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.141447 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.141566 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.142248 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.143103 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.144088 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.144504 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.144627 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.147045 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.147048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.147048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.147059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.148781 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.149194 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.149198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.149198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.149338 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.149545 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.149546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.149546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.149891 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.150035 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.150414 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.150557 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 23:11:46.152062 2177 catalog_manager.cc:1347] Generated new cluster ID: 15a164bf28ff478c9a443f19adcbb553
I20260928 23:11:46.152119 2177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:11:46.161111 2177 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:46.161689 2177 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:46.172398 2177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Generated new TSK 0
I20260928 23:11:46.172572 2177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:46.202158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37417
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:46.279222 2188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:46.279414 2188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:46.279446 2188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:46.280553 2188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:46.280592 2188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:46.280620 2188 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:11:46.280635 2188 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:46.280658 2188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:46.282104 2188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:46.282338 2188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:46.282541 2188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:46.287168 2193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:46.287176 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:46.287272 2196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.287490 2188 server_base.cc:1034] running on GCE node
I20260928 23:11:46.287674 2188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:46.287889 2188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:46.289029 2188 hybrid_clock.cc:648] HybridClock initialized: now 1790637106289012 us; error 28 us; skew 500 ppm
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:46.291363 2188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:46.291760 2188 webserver.cc:460] Webserver started at http://127.0.98.1:33145/ using document root <none> and password file <none>
I20260928 23:11:46.292011 2188 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:46.292069 2188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:46.292192 2188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:46.293186 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "aa796e3ec7d14ed3848f4178ff18a6c3"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "396277708332ff573b924f7e25312776"
server_key_iv: "05be3774f1a18c283823204e5e5d8aa5"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.293586 2188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aa796e3ec7d14ed3848f4178ff18a6c3"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "396277708332ff573b924f7e25312776"
server_key_iv: "05be3774f1a18c283823204e5e5d8aa5"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.294741 2188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:46.295353 2203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.295502 2188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:46.295579 2188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "aa796e3ec7d14ed3848f4178ff18a6c3"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "396277708332ff573b924f7e25312776"
server_key_iv: "05be3774f1a18c283823204e5e5d8aa5"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.295644 2188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:46.318401 2188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:46.320102 2188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:46.320238 2188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:46.320725 2188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:46.320770 2188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.320799 2188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:46.320815 2188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.326892 2188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39029
I20260928 23:11:46.327032 2318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39029 every 8 connection(s)
I20260928 23:11:46.327244 2188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.329988 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.321195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34679 (local address 127.0.98.62:37417)
0928 23:11:46.321258 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.321259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.321485 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:11:46.321985 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.321986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.321997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.322048 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.322843 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.323595 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.325059 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.325395 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.325468 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.327875 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.327876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.327877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.327884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.328913 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.329298 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.329298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.329299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.329346 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.329524 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.329525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.329525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.329619 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.329662 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.329860 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.329924 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:11:46.330164 2212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.321185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.1:34679)
0928 23:11:46.321371 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.321406 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.321888 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.322120 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.322124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.322462 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.322736 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.322743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.323686 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.323688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.324965 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.324968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.325480 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.325481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.325582 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.326093 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.326103 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.327794 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.328982 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.328988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.328990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.329205 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.329391 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.329391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.329392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.329482 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.329662 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.329664 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.329769 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.329843 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.329955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260928 23:11:46.330668 2319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37417
I20260928 23:11:46.330749 2319 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:46.330924 2319 heartbeater.cc:511] Master 127.0.98.62:37417 requested a full tablet report, sending...
I20260928 23:11:46.331453 2116 ts_manager.cc:194] Registered new tserver with Master: aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029)
I20260928 23:11:46.332301 2116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34679
I20260928 23:11:46.336138 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.331678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52595 (local address 127.0.98.62:37417)
0928 23:11:46.331715 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.331716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.331806 (+ 90us) server_negotiation.cc:407] Connection header received
0928 23:11:46.331868 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.331869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.331876 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.331917 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.332215 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.332745 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.333520 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.333690 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.333732 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.334477 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.334478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.334478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.334483 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.335443 (+ 960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.335698 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.335698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.335698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.335737 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.335841 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.335842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.335842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.335920 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.335960 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.336036 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.336085 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:46.336156 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.331624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.1:52595)
0928 23:11:46.331750 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.331761 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.331818 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.331921 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.331922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.332096 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.332161 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.332162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.332818 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.332819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.333465 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.333467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.333730 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.333731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.333778 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.334139 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.334143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.334409 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.335500 (+ 1091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.335501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.335502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.335645 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.335770 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.335770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.335771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.335811 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.335952 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.335952 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.335981 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.336021 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.336083 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260928 23:11:46.337064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2188
I20260928 23:11:46.337148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:46.337324 392 external_mini_cluster.cc:1428] Setting key 13485d5aa918d57d11b865540f1b0d5c
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:46.357858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37417
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:46.434798 2328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:46.434991 2328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:46.435014 2328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:46.436065 2328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:46.436098 2328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:46.436116 2328 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:11:46.436129 2328 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:46.436144 2328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:46.437497 2328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:46.437676 2328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:46.437845 2328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:46.442313 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:46.442332 2336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.442405 2328 server_base.cc:1034] running on GCE node
W20260928 23:11:46.442320 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.442680 2328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:46.442922 2328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:46.444058 2328 hybrid_clock.cc:648] HybridClock initialized: now 1790637106444048 us; error 28 us; skew 500 ppm
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:46.446290 2328 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:46.446667 2328 webserver.cc:460] Webserver started at http://127.0.98.2:34731/ using document root <none> and password file <none>
I20260928 23:11:46.446854 2328 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:46.446897 2328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:46.447014 2328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:46.448017 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "ce98e4bcdcfe24484cef494507ef925f"
server_key_iv: "45db5de98c1a35dfaf25c1152e31d7d1"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.448329 2328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "ce98e4bcdcfe24484cef494507ef925f"
server_key_iv: "45db5de98c1a35dfaf25c1152e31d7d1"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.449621 2328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:46.450278 2343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.450436 2328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:46.450507 2328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "ce98e4bcdcfe24484cef494507ef925f"
server_key_iv: "45db5de98c1a35dfaf25c1152e31d7d1"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.450572 2328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:46.474893 2328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:46.476997 2328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:46.477131 2328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:46.477646 2328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:46.477690 2328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.477721 2328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:46.477744 2328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.483644 2328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42901
I20260928 23:11:46.483788 2458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42901 every 8 connection(s)
I20260928 23:11:46.484006 2328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.486985 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.478129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48033 (local address 127.0.98.62:37417)
0928 23:11:46.478171 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.478173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.478416 (+ 243us) server_negotiation.cc:407] Connection header received
0928 23:11:46.478945 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.478947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.478960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.479030 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.479804 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.480540 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.482047 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.482330 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.482408 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.484711 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.484712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.484712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.484718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.485985 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.486318 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.486319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.486319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.486364 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.486532 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.486533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.486533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.486623 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.486678 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.486871 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.486923 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:11:46.487176 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.478131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.2:48033)
0928 23:11:46.478299 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.478335 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.478832 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.479097 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.479102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.479431 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.479709 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.479715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.480637 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.480638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.481962 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.481965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.482416 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.482417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.482554 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.483081 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.483091 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.484628 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.486053 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.486057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.486059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.486256 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.486404 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.486405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.486405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.486491 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.486663 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.486666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.486781 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.486861 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.486972 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260928 23:11:46.487615 2459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37417
I20260928 23:11:46.487694 2459 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:46.487869 2459 heartbeater.cc:511] Master 127.0.98.62:37417 requested a full tablet report, sending...
I20260928 23:11:46.488219 2115 ts_manager.cc:194] Registered new tserver with Master: 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901)
I20260928 23:11:46.488583 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48033
I20260928 23:11:46.492177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2328
I20260928 23:11:46.492271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:46.492458 392 external_mini_cluster.cc:1428] Setting key e4b2ce96f6d40e6266c5636f2dc5b875
I20260928 23:11:46.493502 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.488545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.2:48289)
0928 23:11:46.488773 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.488784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.488836 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.488968 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.488968 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.489134 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.489183 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.489184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.489799 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.489800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.490293 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.490295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.490603 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.490604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.490682 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.491020 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.491024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.491296 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.492576 (+ 1280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.492577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.492577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.492773 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.492950 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.492950 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.492951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.492997 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.493271 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.493272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.493310 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.493364 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.493434 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":65,"threads_started":1}
I20260928 23:11:46.493546 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.488608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48289 (local address 127.0.98.62:37417)
0928 23:11:46.488646 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.488647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.488828 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:11:46.488910 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.488911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.488921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.488964 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.489268 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.489716 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.490359 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.490548 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.490599 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.491369 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.491370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.491370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.491376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.492508 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.492836 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.492837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.492838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.492893 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.493038 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.493039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.493039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.493196 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.493280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.493403 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.493471 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:46.513511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37417
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:46.590791 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:46.590977 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:46.590999 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:46.592155 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:46.592207 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:46.592227 2468 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 23:11:46.592240 2468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:11:46.592260 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:46.593667 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:46.593852 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:46.594023 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:46.598511 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:46.598521 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.598637 2468 server_base.cc:1034] running on GCE node
W20260928 23:11:46.598521 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:46.598905 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:46.599124 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:46.600255 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790637106600247 us; error 46 us; skew 500 ppm
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:46.602478 2468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:46.602877 2468 webserver.cc:460] Webserver started at http://127.0.98.3:43567/ using document root <none> and password file <none>
I20260928 23:11:46.603072 2468 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:46.603113 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:46.603233 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:46.604171 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ef118ef11934a05b27630106bf19fc4"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "b26ca91784771a3b6a9d8c9b48b8730d"
server_key_iv: "079f2fd5ca610dd3e0e084e455e9c12a"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.604483 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef118ef11934a05b27630106bf19fc4"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "b26ca91784771a3b6a9d8c9b48b8730d"
server_key_iv: "079f2fd5ca610dd3e0e084e455e9c12a"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.605664 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:46.606256 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.606403 2468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:46.606484 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "2ef118ef11934a05b27630106bf19fc4"
format_stamp: "Formatted at 2026-09-28 23:11:46 on dist-test-slave-6vgd"
server_key: "b26ca91784771a3b6a9d8c9b48b8730d"
server_key_iv: "079f2fd5ca610dd3e0e084e455e9c12a"
server_key_version: "encryptionkey@0"
I20260928 23:11:46.606534 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:46.619164 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:46.621052 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:46.621153 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:46.621587 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:46.621630 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.621649 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:46.621660 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.627410 2468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43071
I20260928 23:11:46.627496 2598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43071 every 8 connection(s)
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.627743 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:11:46.630198 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.622008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55465 (local address 127.0.98.62:37417)
0928 23:11:46.622069 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.622072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.622236 (+ 164us) server_negotiation.cc:407] Connection header received
0928 23:11:46.622668 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.622669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.622681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.622749 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.623474 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.624202 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.625633 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.625913 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.625991 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.628213 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.628214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.628214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.628220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.629243 (+ 1023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.629553 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.629553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.629554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.629598 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.629766 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.629767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.629767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.629857 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.629900 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.630086 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.630139 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:11:46.630350 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.621962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.3:55465)
0928 23:11:46.622107 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.622144 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.622587 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.622815 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.622819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.623111 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.623380 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.623387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.624293 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.624294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.625546 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.625548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.625988 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.625988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.626084 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.626523 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.626533 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.628136 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.629303 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.629308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.629310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.629499 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.629641 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.629642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.629642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.629726 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.629898 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.629901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.629998 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.630066 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.630169 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260928 23:11:46.630784 2599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37417
I20260928 23:11:46.630869 2599 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:46.631021 2599 heartbeater.cc:511] Master 127.0.98.62:37417 requested a full tablet report, sending...
I20260928 23:11:46.631363 2115 ts_manager.cc:194] Registered new tserver with Master: 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:46.631804 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55465
I20260928 23:11:46.636214 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.631751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49785 (local address 127.0.98.62:37417)
0928 23:11:46.631783 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.631785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.631890 (+ 105us) server_negotiation.cc:407] Connection header received
0928 23:11:46.631933 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.631934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.631943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.631997 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.632260 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.632708 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.633388 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.633578 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.633628 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.634356 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.634358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.634358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.634364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.635327 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.635595 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.635596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.635596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.635645 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.635798 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.635799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.635799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.635895 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.635946 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.636071 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.636133 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 23:11:46.636238 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.631699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.98.3:49785)
0928 23:11:46.631820 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.631829 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.631887 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.631999 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.632000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.632140 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.632195 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.632196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.632790 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.632791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.633322 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.633324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.633631 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.633632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.633700 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.634028 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.634032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.634282 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.635394 (+ 1112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.635395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.635395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.635539 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.635694 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.635695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.635695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.635748 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.635945 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.635946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.635985 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.636045 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.636134 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260928 23:11:46.637531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2468
I20260928 23:11:46.637606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:46.637753 392 external_mini_cluster.cc:1428] Setting key 9846833dae5d301140b7a6b162925927
I20260928 23:11:46.638399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.658349 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.652533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.0.98.62:37417)
0928 23:11:46.652580 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.652582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.652601 (+ 19us) server_negotiation.cc:407] Connection header received
0928 23:11:46.652622 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.652623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.652631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.652683 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.653248 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.653733 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.654425 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.654613 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.654657 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.656509 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.656510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.656511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.656516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.657597 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.657870 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.657871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.657871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.657913 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.658031 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.658031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.658031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.658115 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.658177 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.658243 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.658291 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:11:46.659749 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:46.661173 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:46.666961 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42901 (local address 127.0.0.1:44388)
0928 23:11:46.663221 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.663262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.663324 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.663954 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.663954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.663956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:46.663986 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.663988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.665132 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.665133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.666022 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.666025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666673 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.666749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.666762 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.666834 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.666885 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":47,"threads_started":1}
I20260928 23:11:46.667012 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.0.1:42832)
0928 23:11:46.663350 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.663359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.663407 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.663657 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.663659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.663661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:46.663721 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.663723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.664677 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.664678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.665951 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.665953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666789 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.666841 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.666844 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.666893 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.666921 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":44,"threads_started":1}
I20260928 23:11:46.667164 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42832 (local address 127.0.98.1:39029)
0928 23:11:46.663512 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.663516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.663525 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:46.663559 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.663560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.663572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.663639 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:46.663845 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.664605 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666010 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666723 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666793 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.666850 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.666905 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.666939 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":79,"threads_started":1}
I20260928 23:11:46.667212 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.98.2:42901)
0928 23:11:46.663771 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.663775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.663783 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:46.663817 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.663818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.663832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.663899 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:46.664051 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.665067 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666088 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666597 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666661 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.666718 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.667080 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.667119 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":77,"threads_started":1}
I20260928 23:11:46.667526 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43071 (local address 127.0.0.1:55016)
0928 23:11:46.663351 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.663359 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.663407 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.663670 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.663671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.663673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:46.663744 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.663746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.664984 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.664985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.666094 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.666096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666960 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.667365 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.667371 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.667422 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.667462 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":38,"thread_start_us":46,"threads_started":1}
I20260928 23:11:46.667762 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.663391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.0.98.3:43071)
0928 23:11:46.663540 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.663544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.663554 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:11:46.663583 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.663585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.663595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.663664 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:46.663820 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.664907 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666150 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.666823 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.666880 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.666928 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.667627 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.667664 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260928 23:11:46.667791 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6bed550a354a7c8823be58e8bc496f (DEFAULT_TABLE table=test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:46.667825 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6bed550a354a7c8823be58e8bc496f (DEFAULT_TABLE table=test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:46.668148 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6bed550a354a7c8823be58e8bc496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.668154 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6bed550a354a7c8823be58e8bc496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.670779 2619 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Bootstrap starting.
I20260928 23:11:46.671096 2620 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Bootstrap starting.
I20260928 23:11:46.671417 2619 tablet_bootstrap.cc:654] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.671623 2620 tablet_bootstrap.cc:654] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.671557 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6bed550a354a7c8823be58e8bc496f (DEFAULT_TABLE table=test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:46.671732 2619 log.cc:824] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:46.671855 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6bed550a354a7c8823be58e8bc496f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.671916 2620 log.cc:824] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:46.672420 2619 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: No bootstrap required, opened a new log
I20260928 23:11:46.672490 2619 ts_tablet_manager.cc:1397] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:46.672564 2620 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: No bootstrap required, opened a new log
I20260928 23:11:46.672621 2620 ts_tablet_manager.cc:1397] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:46.674427 2623 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Bootstrap starting.
I20260928 23:11:46.674434 2620 raft_consensus.cc:348] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.674434 2619 raft_consensus.cc:348] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.674599 2619 raft_consensus.cc:374] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.674599 2620 raft_consensus.cc:374] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.674640 2620 raft_consensus.cc:729] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afd132a3a074b25b54d9202e2b2dcc2, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.674640 2619 raft_consensus.cc:729] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.674750 2619 consensus_queue.cc:260] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.674762 2620 consensus_queue.cc:260] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.675005 2620 ts_tablet_manager.cc:1428] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:11:46.675005 2619 ts_tablet_manager.cc:1428] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:11:46.675129 2319 heartbeater.cc:503] Master 127.0.98.62:37417 was elected leader, sending a full tablet report...
I20260928 23:11:46.675133 2459 heartbeater.cc:503] Master 127.0.98.62:37417 was elected leader, sending a full tablet report...
I20260928 23:11:46.675325 2623 tablet_bootstrap.cc:654] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.675622 2623 log.cc:824] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:46.676302 2623 tablet_bootstrap.cc:492] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: No bootstrap required, opened a new log
I20260928 23:11:46.676352 2623 ts_tablet_manager.cc:1397] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:11:46.677644 2623 raft_consensus.cc:348] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.677767 2623 raft_consensus.cc:374] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.677788 2623 raft_consensus.cc:729] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.677870 2623 consensus_queue.cc:260] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.678068 2623 ts_tablet_manager.cc:1428] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:11:46.678109 2599 heartbeater.cc:503] Master 127.0.98.62:37417 was elected leader, sending a full tablet report...
W20260928 23:11:46.734575 2460 tablet.cc:2369] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:11:46.828063 2320 tablet.cc:2369] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:11:46.850440 2627 raft_consensus.cc:482] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:46.850553 2627 raft_consensus.cc:504] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.850895 2627 leader_election.cc:288] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029)
I20260928 23:11:46.853775 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.851116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60665 (local address 127.0.98.1:39029)
0928 23:11:46.851160 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.851162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.851206 (+ 44us) server_negotiation.cc:407] Connection header received
0928 23:11:46.851281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.851282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.851295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.851357 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:46.851535 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.852085 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853040 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.853462 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853516 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.853526 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.853666 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.853706 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:11:46.853792 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.851138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57115 (local address 127.0.98.2:42901)
0928 23:11:46.851175 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.851177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.851322 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:11:46.851365 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.851366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.851377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.851440 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:46.851599 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.852148 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853068 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.853477 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853541 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.853556 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.853694 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.853735 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:11:46.853790 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.851064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.98.3:60665)
0928 23:11:46.851141 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.851152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.851216 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.851392 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.851393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.851394 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:46.851482 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.851484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.852154 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.852155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.852961 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.852964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.853528 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.853601 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.853605 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.853650 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.853690 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 23:11:46.853813 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.851064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42901 (local address 127.0.98.3:57115)
0928 23:11:46.851231 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.851241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.851309 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.851442 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.851443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.851444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:46.851536 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.851538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.852220 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.852221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.852984 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.852987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.853553 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.853554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.853626 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.853630 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.853687 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.853721 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":13,"thread_start_us":93,"threads_started":1}
I20260928 23:11:46.854158 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6bed550a354a7c8823be58e8bc496f" candidate_uuid: "2ef118ef11934a05b27630106bf19fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" is_pre_election: true
I20260928 23:11:46.854151 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6bed550a354a7c8823be58e8bc496f" candidate_uuid: "2ef118ef11934a05b27630106bf19fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" is_pre_election: true
I20260928 23:11:46.854305 2413 raft_consensus.cc:2454] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef118ef11934a05b27630106bf19fc4 in term 0.
I20260928 23:11:46.854305 2273 raft_consensus.cc:2454] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef118ef11934a05b27630106bf19fc4 in term 0.
I20260928 23:11:46.854491 2485 leader_election.cc:302] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef118ef11934a05b27630106bf19fc4, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:46.854614 2627 raft_consensus.cc:2781] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:46.854676 2627 raft_consensus.cc:482] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:46.854692 2627 raft_consensus.cc:3037] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:46.855569 2627 raft_consensus.cc:504] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.855684 2627 leader_election.cc:288] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [CANDIDATE]: Term 1 election: Requested vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029)
I20260928 23:11:46.855809 2273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6bed550a354a7c8823be58e8bc496f" candidate_uuid: "2ef118ef11934a05b27630106bf19fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3"
I20260928 23:11:46.855877 2273 raft_consensus.cc:3037] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:46.855890 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6bed550a354a7c8823be58e8bc496f" candidate_uuid: "2ef118ef11934a05b27630106bf19fc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
I20260928 23:11:46.855948 2413 raft_consensus.cc:3037] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:46.856679 2273 raft_consensus.cc:2454] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef118ef11934a05b27630106bf19fc4 in term 1.
I20260928 23:11:46.856679 2413 raft_consensus.cc:2454] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef118ef11934a05b27630106bf19fc4 in term 1.
I20260928 23:11:46.856834 2486 leader_election.cc:302] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, 2ef118ef11934a05b27630106bf19fc4, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:46.856922 2627 raft_consensus.cc:2781] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:46.857036 2627 raft_consensus.cc:686] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 1 LEADER]: Becoming Leader. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Running, Role: LEADER
I20260928 23:11:46.857116 2627 consensus_queue.cc:237] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.857915 2115 catalog_manager.cc:5495] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ef118ef11934a05b27630106bf19fc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } health_report { overall_health: HEALTHY } } }
W20260928 23:11:46.878278 2600 tablet.cc:2369] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 23:11:46 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:46.915585 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.909797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37417 (local address 127.0.0.1:42850)
0928 23:11:46.909945 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:46.909955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:46.910017 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:46.910121 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:46.910121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:46.910416 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:46.910461 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.910463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.911039 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.911039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.911581 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:46.911583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.911860 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.911860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:46.911896 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.912322 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:46.912326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:46.913620 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:46.914774 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.914780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.914782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.914961 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.915108 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:46.915109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:46.915109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:46.915158 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:46.915324 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:46.915327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:46.915372 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:11:46.915416 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:11:46.915497 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260928 23:11:46.915629 2165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:46.909836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.0.98.62:37417)
0928 23:11:46.909909 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:46.909912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:46.910012 (+ 100us) server_negotiation.cc:407] Connection header received
0928 23:11:46.910067 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:46.910068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:46.910076 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:46.910112 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:46.910512 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.910989 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.911638 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:46.911814 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:46.911857 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:46.913683 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:46.913684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:46.913684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:46.913690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:46.914706 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.915016 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.915017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.915017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.915063 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:46.915196 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:46.915196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:46.915196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:46.915285 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:46.915328 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:46.915437 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:11:46.915537 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":47}
I20260928 23:11:46.917475 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42850:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:11:46.917650 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:46.918993 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 289dabb554c1492abb1044674eef7d1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b86dbccb9354a938a5119f8bdb76dcd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:46.919001 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 289dabb554c1492abb1044674eef7d1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b86dbccb9354a938a5119f8bdb76dcd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:46.919132 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289dabb554c1492abb1044674eef7d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.919139 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289dabb554c1492abb1044674eef7d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.920162 2620 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: Bootstrap starting.
I20260928 23:11:46.920181 2623 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: Bootstrap starting.
I20260928 23:11:46.920281 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 289dabb554c1492abb1044674eef7d1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b86dbccb9354a938a5119f8bdb76dcd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:11:46.920425 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289dabb554c1492abb1044674eef7d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:46.920581 2623 tablet_bootstrap.cc:654] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.920815 2620 tablet_bootstrap.cc:654] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.921200 2623 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: No bootstrap required, opened a new log
I20260928 23:11:46.921273 2623 ts_tablet_manager.cc:1397] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:11:46.921403 2620 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: No bootstrap required, opened a new log
I20260928 23:11:46.921401 2623 raft_consensus.cc:348] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.921442 2620 ts_tablet_manager.cc:1397] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:46.921455 2623 raft_consensus.cc:374] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.921468 2623 raft_consensus.cc:729] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.921502 2623 consensus_queue.cc:260] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.921581 2623 ts_tablet_manager.cc:1428] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.921572 2620 raft_consensus.cc:348] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.921633 2620 raft_consensus.cc:374] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.921607 2627 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:46.921651 2620 raft_consensus.cc:729] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afd132a3a074b25b54d9202e2b2dcc2, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.921667 2627 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:46.921694 2620 consensus_queue.cc:260] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.921780 2620 ts_tablet_manager.cc:1428] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.921938 2625 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:46.922030 2625 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:46.922109 2619 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: Bootstrap starting.
I20260928 23:11:46.922449 2619 tablet_bootstrap.cc:654] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:46.922981 2619 tablet_bootstrap.cc:492] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: No bootstrap required, opened a new log
I20260928 23:11:46.923020 2619 ts_tablet_manager.cc:1397] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:46.923149 2619 raft_consensus.cc:348] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.923199 2619 raft_consensus.cc:374] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:46.923215 2619 raft_consensus.cc:729] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Initialized, Role: FOLLOWER
I20260928 23:11:46.923255 2619 consensus_queue.cc:260] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:46.923357 2619 ts_tablet_manager.cc:1428] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:46.923413 2624 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:46.923506 2624 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:47.086802 2624 raft_consensus.cc:482] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:47.086905 2624 raft_consensus.cc:504] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.087216 2624 leader_election.cc:288] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:47.090281 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.087410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42901 (local address 127.0.98.1:38369)
0928 23:11:47.087611 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.087619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.087681 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.087802 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.087803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.087805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:47.087870 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.087872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.088457 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.088458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.089390 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.089392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.090059 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.090060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.090121 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.090126 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.090151 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.090188 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":25,"thread_start_us":62,"threads_started":1}
I20260928 23:11:47.090284 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.087441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38369 (local address 127.0.98.2:42901)
0928 23:11:47.087486 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.087488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.087683 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:11:47.087737 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.087738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.087747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.087796 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:47.087926 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.088390 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.089461 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.089992 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.090172 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.090182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.090198 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.090230 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:47.090651 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289dabb554c1492abb1044674eef7d1b" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" is_pre_election: true
I20260928 23:11:47.090755 2413 raft_consensus.cc:2454] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:47.090808 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.087408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43071 (local address 127.0.98.1:60991)
0928 23:11:47.087533 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.087544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.087601 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.087734 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.087735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.087737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:11:47.087816 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.087819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.088437 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.088438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.089949 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.089951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.090591 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.090592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.090659 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.090663 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.090707 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.090734 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":48,"thread_start_us":68,"threads_started":1}
I20260928 23:11:47.090843 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.087534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60991 (local address 127.0.98.3:43071)
0928 23:11:47.087582 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.087584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.087596 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:11:47.087653 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.087654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.087666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.087723 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:11:47.087879 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.088367 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.090040 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.090519 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.090586 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.090595 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.090717 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.090766 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:47.091019 2206 leader_election.cc:302] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:47.091176 2624 raft_consensus.cc:2781] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:47.091226 2624 raft_consensus.cc:482] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:47.091244 2624 raft_consensus.cc:3037] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.091256 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289dabb554c1492abb1044674eef7d1b" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4" is_pre_election: true
I20260928 23:11:47.091389 2553 raft_consensus.cc:2454] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:47.139034 2624 raft_consensus.cc:504] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.139257 2624 leader_election.cc:288] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:47.139914 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289dabb554c1492abb1044674eef7d1b" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
I20260928 23:11:47.140002 2413 raft_consensus.cc:3037] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.140798 2413 raft_consensus.cc:2454] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:47.141053 2206 leader_election.cc:302] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:47.141160 2624 raft_consensus.cc:2781] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:47.141306 2624 raft_consensus.cc:686] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Running, Role: LEADER
I20260928 23:11:47.141408 2624 consensus_queue.cc:237] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.141850 2642 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: TxnStatusTablet state changed. Reason: New leader aa796e3ec7d14ed3848f4178ff18a6c3. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.141989 2642 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:47.142169 2115 catalog_manager.cc:5495] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } health_report { overall_health: UNKNOWN } } }
I20260928 23:11:47.142590 2644 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:11:47.142705 2644 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:11:47.143853 2553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289dabb554c1492abb1044674eef7d1b" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4"
I20260928 23:11:47.143960 2553 raft_consensus.cc:3037] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.149945 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.147628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.0.98.1:39029)
0928 23:11:47.147673 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.147675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.147683 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:47.147731 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.147732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.147748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.147800 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.147898 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.148434 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.149163 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.149428 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.149484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.149659 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.149753 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.149815 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.149857 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.149878 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:11:47.149959 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.147548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.0.1:42848)
0928 23:11:47.147610 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.147620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.147663 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.147803 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.147804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.147805 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:47.147852 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.147855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.148500 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.148501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.149087 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.149089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.149488 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.149488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.149552 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.149569 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.149804 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.149812 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.149855 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.149898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:11:47.151587 2413 raft_consensus.cc:1261] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.151887 2624 consensus_queue.cc:1035] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:47.153069 2625 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: TxnStatusTablet state changed. Reason: New leader aa796e3ec7d14ed3848f4178ff18a6c3. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.153163 2625 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:47.153867 2647 mvcc.cc:205] Tried to move back new op lower bound from 7334449590890049536 to 7334449590851518464. Current Snapshot: MvccSnapshot[applied={T|T < 7334449590890049536}]
I20260928 23:11:47.153960 2642 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.154033 2642 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:47.154453 2625 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.154537 2625 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 0afd132a3a074b25b54d9202e2b2dcc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:47.158164 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.155939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.0.98.3:43071)
0928 23:11:47.155980 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.155982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.155989 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:11:47.156022 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.156023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.156040 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.156096 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.156194 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.156666 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.157422 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.157624 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.157684 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.157864 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.157965 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.158027 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.158072 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.158095 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:11:47.162411 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.160302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59339 (local address 127.0.98.1:39029)
0928 23:11:47.160339 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.160341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.160394 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:11:47.160450 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.160451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.160460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.160499 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.160610 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.161062 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.161813 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.162006 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.162056 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.162205 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.162236 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.162283 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.162331 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.162348 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:11:47.162446 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.160251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.98.3:59339)
0928 23:11:47.160343 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.160353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.160407 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.160510 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.160511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.160512 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:47.160562 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.160563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.161128 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.161129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.161738 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.161739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.162060 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.162061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.162118 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.162132 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.162289 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.162291 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.162330 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.162368 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":46,"threads_started":1}
I20260928 23:11:47.164544 2413 raft_consensus.cc:1261] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Refusing update from remote peer 2ef118ef11934a05b27630106bf19fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.164602 2273 raft_consensus.cc:1261] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Refusing update from remote peer 2ef118ef11934a05b27630106bf19fc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.164798 2627 consensus_queue.cc:1035] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:47.164932 2627 consensus_queue.cc:1035] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:47.166729 2657 mvcc.cc:205] Tried to move back new op lower bound from 7334449590944358400 to 7334449589687304192. Current Snapshot: MvccSnapshot[applied={T|T < 7334449590944358400}]
I20260928 23:11:47.176447 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.173590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38199 (local address 127.0.98.3:43071)
0928 23:11:47.173623 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.173625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.173769 (+ 144us) server_negotiation.cc:407] Connection header received
0928 23:11:47.173828 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.173829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.173838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.173879 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.174015 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.174464 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.175658 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.175969 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.176010 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.176230 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.176271 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.176312 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.176378 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.176395 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 23:11:47.176518 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.173544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43071 (local address 127.0.98.1:38199)
0928 23:11:47.173709 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.173720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.173775 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.173891 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.173892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.173893 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:47.173963 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.173965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.174527 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.174527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.175514 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.175516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.176081 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.176082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.176146 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.176161 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.176326 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.176328 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.176375 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.176421 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":51,"threads_started":1}
I20260928 23:11:47.182883 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:47.182979 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:47.183619 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Sending DeleteTablet for 3 replicas of tablet 8a6bed550a354a7c8823be58e8bc496f
I20260928 23:11:47.183892 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6bed550a354a7c8823be58e8bc496f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44388
I20260928 23:11:47.183950 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6bed550a354a7c8823be58e8bc496f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55016
I20260928 23:11:47.184072 2666 tablet_replica.cc:330] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: stopping tablet replica
I20260928 23:11:47.184111 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6bed550a354a7c8823be58e8bc496f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260928 23:11:47.184165 2666 raft_consensus.cc:2229] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:47.184217 2666 raft_consensus.cc:2258] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:47.184566 2666 ts_tablet_manager.cc:1905] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:47.184665 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:47.184782 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:47.185976 2667 tablet_replica.cc:330] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: stopping tablet replica
I20260928 23:11:47.186024 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e (DEFAULT_TABLE table=test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:47.186167 2668 tablet_replica.cc:330] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: stopping tablet replica
I20260928 23:11:47.186293 2253 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e (DEFAULT_TABLE table=test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:47.186401 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70d24b0bac4fcb8dff2ed5beca744e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:47.186493 2667 raft_consensus.cc:2229] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:11:47.186316 2668 raft_consensus.cc:2229] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:47.186641 2668 raft_consensus.cc:2258] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:47.186762 2667 raft_consensus.cc:2258] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:47.187144 2667 ts_tablet_manager.cc:1905] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:47.187758 2619 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Bootstrap starting.
I20260928 23:11:47.187805 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e (DEFAULT_TABLE table=test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:47.187903 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70d24b0bac4fcb8dff2ed5beca744e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:47.188349 2619 tablet_bootstrap.cc:654] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:47.188773 2668 ts_tablet_manager.cc:1905] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:47.193538 2619 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: No bootstrap required, opened a new log
I20260928 23:11:47.193586 2619 ts_tablet_manager.cc:1397] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:11:47.193723 2619 raft_consensus.cc:348] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.193789 2619 raft_consensus.cc:374] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:47.193807 2619 raft_consensus.cc:729] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Initialized, Role: FOLLOWER
I20260928 23:11:47.193849 2619 consensus_queue.cc:260] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.193924 2619 ts_tablet_manager.cc:1428] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:47.229513 2553 raft_consensus.cc:2454] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:47.229874 2552 raft_consensus.cc:1261] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.231218 2623 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Bootstrap starting.
I20260928 23:11:47.231669 2623 tablet_bootstrap.cc:654] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:47.234123 2642 consensus_queue.cc:1035] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:47.234299 2623 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: No bootstrap required, opened a new log
I20260928 23:11:47.234338 2623 ts_tablet_manager.cc:1397] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 23:11:47.234478 2623 raft_consensus.cc:348] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.234540 2623 raft_consensus.cc:374] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:47.234558 2623 raft_consensus.cc:729] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Initialized, Role: FOLLOWER
I20260928 23:11:47.234594 2623 consensus_queue.cc:260] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.235140 2627 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.235216 2627 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:47.235284 2660 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: TxnStatusTablet state changed. Reason: New leader aa796e3ec7d14ed3848f4178ff18a6c3. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.235325 2660 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P 2ef118ef11934a05b27630106bf19fc4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:11:47.236296 2623 ts_tablet_manager.cc:1428] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 23:11:47.236685 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70d24b0bac4fcb8dff2ed5beca744e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:47.237674 2620 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Bootstrap starting.
I20260928 23:11:47.238065 2620 tablet_bootstrap.cc:654] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:47.238686 2620 tablet_bootstrap.cc:492] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: No bootstrap required, opened a new log
I20260928 23:11:47.238730 2620 ts_tablet_manager.cc:1397] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:47.238857 2620 raft_consensus.cc:348] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.238911 2620 raft_consensus.cc:374] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:47.238929 2620 raft_consensus.cc:729] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afd132a3a074b25b54d9202e2b2dcc2, State: Initialized, Role: FOLLOWER
I20260928 23:11:47.238971 2620 consensus_queue.cc:260] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.239049 2620 ts_tablet_manager.cc:1428] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:47.277259 2668 ts_tablet_manager.cc:1918] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:47.277384 2668 log.cc:1196] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/8a6bed550a354a7c8823be58e8bc496f
I20260928 23:11:47.277631 2668 ts_tablet_manager.cc:1939] T 8a6bed550a354a7c8823be58e8bc496f P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting consensus metadata
I20260928 23:11:47.301025 2101 catalog_manager.cc:4841] TS aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029): tablet 8a6bed550a354a7c8823be58e8bc496f (table test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]) successfully deleted
I20260928 23:11:47.308573 2667 ts_tablet_manager.cc:1918] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:47.308686 2667 log.cc:1196] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/8a6bed550a354a7c8823be58e8bc496f
I20260928 23:11:47.308934 2667 ts_tablet_manager.cc:1939] T 8a6bed550a354a7c8823be58e8bc496f P 2ef118ef11934a05b27630106bf19fc4: Deleting consensus metadata
I20260928 23:11:47.309659 2101 catalog_manager.cc:4841] TS 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071): tablet 8a6bed550a354a7c8823be58e8bc496f (table test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]) successfully deleted
I20260928 23:11:47.309930 2666 ts_tablet_manager.cc:1918] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:47.310065 2666 log.cc:1196] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/8a6bed550a354a7c8823be58e8bc496f
I20260928 23:11:47.310473 2666 ts_tablet_manager.cc:1939] T 8a6bed550a354a7c8823be58e8bc496f P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting consensus metadata
I20260928 23:11:47.334137 2102 catalog_manager.cc:4841] TS 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901): tablet 8a6bed550a354a7c8823be58e8bc496f (table test-table [id=b55b8a7c491d49e6bb4c64b13ded7524]) successfully deleted
I20260928 23:11:47.342118 2642 raft_consensus.cc:482] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:47.342231 2642 raft_consensus.cc:504] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.342387 2642 leader_election.cc:288] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:47.342618 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70d24b0bac4fcb8dff2ed5beca744e" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4" is_pre_election: true
I20260928 23:11:47.342620 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70d24b0bac4fcb8dff2ed5beca744e" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" is_pre_election: true
I20260928 23:11:47.342705 2552 raft_consensus.cc:2454] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:47.342729 2413 raft_consensus.cc:2454] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:47.342923 2205 leader_election.cc:302] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef118ef11934a05b27630106bf19fc4, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:47.343011 2642 raft_consensus.cc:2781] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:47.343042 2642 raft_consensus.cc:482] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:47.343060 2642 raft_consensus.cc:3037] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.384018 2642 raft_consensus.cc:504] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.384209 2642 leader_election.cc:288] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:47.384461 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70d24b0bac4fcb8dff2ed5beca744e" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
I20260928 23:11:47.384460 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70d24b0bac4fcb8dff2ed5beca744e" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4"
I20260928 23:11:47.384546 2413 raft_consensus.cc:3037] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.384560 2552 raft_consensus.cc:3037] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:47.385139 2413 raft_consensus.cc:2454] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:47.385355 2206 leader_election.cc:302] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:47.385375 2552 raft_consensus.cc:2454] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:47.385458 2642 raft_consensus.cc:2781] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:47.385510 2642 raft_consensus.cc:686] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Running, Role: LEADER
I20260928 23:11:47.385550 2642 consensus_queue.cc:237] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:47.386090 2114 catalog_manager.cc:5495] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } health_report { overall_health: UNKNOWN } } }
I20260928 23:11:47.417328 2642 tablet_replica.cc:439] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } } }
I20260928 23:11:47.417461 2642 tablet_replica.cc:442] T 289dabb554c1492abb1044674eef7d1b P aa796e3ec7d14ed3848f4178ff18a6c3: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:11:47.420651 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.418796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.0.98.1:39029)
0928 23:11:47.418849 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.418851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.418858 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:11:47.418876 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.418877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.418884 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.418928 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.419033 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.419495 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.420154 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.420330 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.420379 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.420466 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.420491 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.420526 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.420563 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.420580 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:11:47.424309 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.422555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.98.1:57381)
0928 23:11:47.422604 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:47.422614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:47.422656 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:47.422745 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:47.422746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:47.422746 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:11:47.422775 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.422776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.423284 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.423285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.423806 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:47.423807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.424064 (+ 257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.424064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:47.424097 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.424101 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.424193 (+ 92us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.424194 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:11:47.424221 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 23:11:47.424254 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:11:47.424315 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:47.422608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57381 (local address 127.0.98.1:39029)
0928 23:11:47.422650 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:47.422652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:47.422660 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:47.422693 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:47.422693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:47.422700 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:47.422738 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:47.422809 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.423227 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.423859 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:47.424020 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:47.424060 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:47.424142 (+ 82us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:47.424163 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:47.424192 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:47.424222 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 23:11:47.424234 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:11:47.425900 2552 raft_consensus.cc:1261] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.425920 2413 raft_consensus.cc:1261] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:47.426090 2642 consensus_queue.cc:1035] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:47.426198 2675 consensus_queue.cc:1035] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:11:47.430233 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:11:48.952709 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:48.952805 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:48.953644 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Sending DeleteTablet for 3 replicas of tablet 8d70d24b0bac4fcb8dff2ed5beca744e
I20260928 23:11:48.953902 2253 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260928 23:11:48.953941 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44388
I20260928 23:11:48.954092 2700 tablet_replica.cc:330] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: stopping tablet replica
I20260928 23:11:48.954087 2699 tablet_replica.cc:330] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: stopping tablet replica
I20260928 23:11:48.954152 2700 raft_consensus.cc:2229] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:48.954202 2700 raft_consensus.cc:2258] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:48.954180 2699 raft_consensus.cc:2229] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:11:48.954299 2699 raft_consensus.cc:2258] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:48.954447 2700 ts_tablet_manager.cc:1905] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:48.954793 2699 ts_tablet_manager.cc:1905] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:48.955575 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70d24b0bac4fcb8dff2ed5beca744e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55016
I20260928 23:11:48.956303 2699 ts_tablet_manager.cc:1918] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:11:48.956319 2700 ts_tablet_manager.cc:1918] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:11:48.956359 2699 log.cc:1196] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/8d70d24b0bac4fcb8dff2ed5beca744e
I20260928 23:11:48.956370 2700 log.cc:1196] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/8d70d24b0bac4fcb8dff2ed5beca744e
I20260928 23:11:48.956585 2699 ts_tablet_manager.cc:1939] T 8d70d24b0bac4fcb8dff2ed5beca744e P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting consensus metadata
I20260928 23:11:48.956588 2700 ts_tablet_manager.cc:1939] T 8d70d24b0bac4fcb8dff2ed5beca744e P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting consensus metadata
I20260928 23:11:48.956614 2702 tablet_replica.cc:330] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: stopping tablet replica
I20260928 23:11:48.956658 2702 raft_consensus.cc:2229] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:48.956703 2702 raft_consensus.cc:2258] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:48.956898 2702 ts_tablet_manager.cc:1905] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:48.956920 2101 catalog_manager.cc:4841] TS aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029): tablet 8d70d24b0bac4fcb8dff2ed5beca744e (table test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]) successfully deleted
I20260928 23:11:48.957008 2102 catalog_manager.cc:4841] TS 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901): tablet 8d70d24b0bac4fcb8dff2ed5beca744e (table test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]) successfully deleted
I20260928 23:11:48.958125 2702 ts_tablet_manager.cc:1918] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:11:48.958169 2702 log.cc:1196] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/8d70d24b0bac4fcb8dff2ed5beca744e
I20260928 23:11:48.958329 2702 ts_tablet_manager.cc:1939] T 8d70d24b0bac4fcb8dff2ed5beca744e P 2ef118ef11934a05b27630106bf19fc4: Deleting consensus metadata
I20260928 23:11:48.958792 2101 catalog_manager.cc:4841] TS 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071): tablet 8d70d24b0bac4fcb8dff2ed5beca744e (table test-table [id=e8bd40347b3d43a3b5bf152c7bdfe773]) successfully deleted
Sep 28 23:11:48 dist-test-slave-6vgd krb5kdc[2071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:11:48 dist-test-slave-6vgd krb5kdc[2071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:48.975211 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:48.968781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.0.98.62:37417)
0928 23:11:48.968910 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:48.968913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:48.968921 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:48.968984 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:48.968985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:48.968996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:48.969054 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:48.969618 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:48.970130 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:48.970830 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:48.971030 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:48.971084 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:48.973068 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:48.973069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:48.973069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:48.973076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:48.974224 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:48.974544 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:48.974546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:48.974546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:48.974596 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:48.974748 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:48.974748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:48.974749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:48.974848 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:48.974943 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:48.975039 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 23:11:48.975115 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 23:11:48.976370 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:11:48.976507 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:48.977850 2393 tablet_service.cc:1467] Processing CreateTablet for tablet d02da3ae5b914561899a48ac7d4150f6 (DEFAULT_TABLE table=test-table [id=92100914f4074fca8ad87afd4d07ca84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:48.977854 2533 tablet_service.cc:1467] Processing CreateTablet for tablet d02da3ae5b914561899a48ac7d4150f6 (DEFAULT_TABLE table=test-table [id=92100914f4074fca8ad87afd4d07ca84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:48.977929 2253 tablet_service.cc:1467] Processing CreateTablet for tablet d02da3ae5b914561899a48ac7d4150f6 (DEFAULT_TABLE table=test-table [id=92100914f4074fca8ad87afd4d07ca84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:48.977967 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02da3ae5b914561899a48ac7d4150f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:48.977996 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02da3ae5b914561899a48ac7d4150f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:48.978026 2253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02da3ae5b914561899a48ac7d4150f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:48.978866 2710 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Bootstrap starting.
I20260928 23:11:48.979274 2711 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Bootstrap starting.
I20260928 23:11:48.979280 2710 tablet_bootstrap.cc:654] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:48.979616 2711 tablet_bootstrap.cc:654] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:48.979820 2710 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: No bootstrap required, opened a new log
I20260928 23:11:48.979871 2710 ts_tablet_manager.cc:1397] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:48.979892 2712 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Bootstrap starting.
I20260928 23:11:48.979984 2710 raft_consensus.cc:348] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.980072 2710 raft_consensus.cc:374] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:48.980088 2710 raft_consensus.cc:729] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afd132a3a074b25b54d9202e2b2dcc2, State: Initialized, Role: FOLLOWER
I20260928 23:11:48.980150 2711 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: No bootstrap required, opened a new log
I20260928 23:11:48.980154 2710 consensus_queue.cc:260] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.980195 2712 tablet_bootstrap.cc:654] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:48.980201 2711 ts_tablet_manager.cc:1397] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:48.980307 2710 ts_tablet_manager.cc:1428] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:48.980310 2711 raft_consensus.cc:348] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.980412 2711 raft_consensus.cc:374] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:48.980446 2711 raft_consensus.cc:729] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Initialized, Role: FOLLOWER
I20260928 23:11:48.980489 2711 consensus_queue.cc:260] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.980693 2711 ts_tablet_manager.cc:1428] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:48.980700 2712 tablet_bootstrap.cc:492] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: No bootstrap required, opened a new log
I20260928 23:11:48.980741 2712 ts_tablet_manager.cc:1397] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:48.980832 2712 raft_consensus.cc:348] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.980906 2712 raft_consensus.cc:374] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:48.980926 2712 raft_consensus.cc:729] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Initialized, Role: FOLLOWER
I20260928 23:11:48.980962 2712 consensus_queue.cc:260] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:48.981119 2712 ts_tablet_manager.cc:1428] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:49.061014 2687 raft_consensus.cc:482] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:49.061110 2687 raft_consensus.cc:504] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.061293 2687 leader_election.cc:288] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:49.061507 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02da3ae5b914561899a48ac7d4150f6" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" is_pre_election: true
I20260928 23:11:49.061532 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02da3ae5b914561899a48ac7d4150f6" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4" is_pre_election: true
I20260928 23:11:49.061625 2552 raft_consensus.cc:2454] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:49.061626 2413 raft_consensus.cc:2454] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:49.061795 2206 leader_election.cc:302] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:49.061915 2687 raft_consensus.cc:2781] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:49.061968 2687 raft_consensus.cc:482] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:49.061990 2687 raft_consensus.cc:3037] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.062660 2687 raft_consensus.cc:504] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.062778 2687 leader_election.cc:288] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:49.062924 2413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02da3ae5b914561899a48ac7d4150f6" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
I20260928 23:11:49.062935 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02da3ae5b914561899a48ac7d4150f6" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4"
I20260928 23:11:49.063000 2552 raft_consensus.cc:3037] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.063000 2413 raft_consensus.cc:3037] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.063563 2413 raft_consensus.cc:2454] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:49.063611 2552 raft_consensus.cc:2454] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:49.063731 2206 leader_election.cc:302] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:49.063817 2687 raft_consensus.cc:2781] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:49.063859 2687 raft_consensus.cc:686] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Running, Role: LEADER
I20260928 23:11:49.063905 2687 consensus_queue.cc:237] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.064481 2115 catalog_manager.cc:5495] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } health_report { overall_health: UNKNOWN } } }
I20260928 23:11:49.073987 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:49.071773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.0.98.1:39029)
0928 23:11:49.071921 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:49.071923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:49.071929 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:11:49.071959 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:49.071959 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:49.071968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:49.072021 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:11:49.072129 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:49.072614 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:49.073385 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:49.073588 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:49.073641 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:49.073757 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:11:49.073792 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:11:49.073833 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:49.073878 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 23:11:49.073898 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 23:11:49.075701 2552 raft_consensus.cc:1261] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:49.075760 2412 raft_consensus.cc:1261] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:49.075963 2682 consensus_queue.cc:1035] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:49.076071 2624 consensus_queue.cc:1035] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:49.077400 2684 mvcc.cc:205] Tried to move back new op lower bound from 7334449598772981760 to 7334449598725988352. Current Snapshot: MvccSnapshot[applied={T|T < 7334449598772981760}]
I20260928 23:11:49.084370 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:49.084481 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:49.085017 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Sending DeleteTablet for 3 replicas of tablet d02da3ae5b914561899a48ac7d4150f6
I20260928 23:11:49.085224 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet d02da3ae5b914561899a48ac7d4150f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260928 23:11:49.085268 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet d02da3ae5b914561899a48ac7d4150f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44388
I20260928 23:11:49.085315 2699 tablet_replica.cc:330] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: stopping tablet replica
I20260928 23:11:49.085366 2699 raft_consensus.cc:2229] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:11:49.085371 2700 tablet_replica.cc:330] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: stopping tablet replica
I20260928 23:11:49.085358 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet d02da3ae5b914561899a48ac7d4150f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55016
I20260928 23:11:49.085414 2700 raft_consensus.cc:2229] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:49.085419 2699 raft_consensus.cc:2258] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:49.085431 2702 tablet_replica.cc:330] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: stopping tablet replica
I20260928 23:11:49.085467 2700 raft_consensus.cc:2258] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:49.085477 2702 raft_consensus.cc:2229] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:49.085508 2702 raft_consensus.cc:2258] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:49.085654 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 23:11:49.085760 2699 ts_tablet_manager.cc:1905] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 23:11:49.085790 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:11:49.085801 2702 ts_tablet_manager.cc:1905] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:49.086072 2700 ts_tablet_manager.cc:1905] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:49.087376 2700 ts_tablet_manager.cc:1918] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:49.087417 2702 ts_tablet_manager.cc:1918] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:49.087432 2700 log.cc:1196] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/d02da3ae5b914561899a48ac7d4150f6
I20260928 23:11:49.087464 2702 log.cc:1196] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/d02da3ae5b914561899a48ac7d4150f6
I20260928 23:11:49.087642 2702 ts_tablet_manager.cc:1939] T d02da3ae5b914561899a48ac7d4150f6 P 2ef118ef11934a05b27630106bf19fc4: Deleting consensus metadata
I20260928 23:11:49.087653 2700 ts_tablet_manager.cc:1939] T d02da3ae5b914561899a48ac7d4150f6 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting consensus metadata
I20260928 23:11:49.087893 2101 catalog_manager.cc:4841] TS 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071): tablet d02da3ae5b914561899a48ac7d4150f6 (table test-table [id=92100914f4074fca8ad87afd4d07ca84]) successfully deleted
I20260928 23:11:49.087909 2102 catalog_manager.cc:4841] TS 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901): tablet d02da3ae5b914561899a48ac7d4150f6 (table test-table [id=92100914f4074fca8ad87afd4d07ca84]) successfully deleted
I20260928 23:11:49.088140 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 474f19214c4c4da88723697eac8f2063 (DEFAULT_TABLE table=test-table [id=708c640fd08747baa025d831f5352d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:49.088138 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 474f19214c4c4da88723697eac8f2063 (DEFAULT_TABLE table=test-table [id=708c640fd08747baa025d831f5352d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:49.088140 2393 tablet_service.cc:1467] Processing CreateTablet for tablet 474f19214c4c4da88723697eac8f2063 (DEFAULT_TABLE table=test-table [id=708c640fd08747baa025d831f5352d36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:11:49.088208 2699 ts_tablet_manager.cc:1918] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:11:49.088234 2393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474f19214c4c4da88723697eac8f2063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:49.088234 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474f19214c4c4da88723697eac8f2063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:49.088234 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474f19214c4c4da88723697eac8f2063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:49.088251 2699 log.cc:1196] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/d02da3ae5b914561899a48ac7d4150f6
I20260928 23:11:49.088459 2699 ts_tablet_manager.cc:1939] T d02da3ae5b914561899a48ac7d4150f6 P aa796e3ec7d14ed3848f4178ff18a6c3: Deleting consensus metadata
I20260928 23:11:49.089325 2710 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Bootstrap starting.
I20260928 23:11:49.089421 2711 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3: Bootstrap starting.
I20260928 23:11:49.089522 2101 catalog_manager.cc:4841] TS aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029): tablet d02da3ae5b914561899a48ac7d4150f6 (table test-table [id=92100914f4074fca8ad87afd4d07ca84]) successfully deleted
I20260928 23:11:49.089751 2711 tablet_bootstrap.cc:654] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:49.089846 2710 tablet_bootstrap.cc:654] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:49.090135 2712 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Bootstrap starting.
I20260928 23:11:49.090247 2711 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3: No bootstrap required, opened a new log
I20260928 23:11:49.090291 2711 ts_tablet_manager.cc:1397] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:49.090384 2710 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: No bootstrap required, opened a new log
I20260928 23:11:49.090394 2712 tablet_bootstrap.cc:654] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:49.090368 2711 raft_consensus.cc:348] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.090420 2711 raft_consensus.cc:374] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:49.090432 2711 raft_consensus.cc:729] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Initialized, Role: FOLLOWER
I20260928 23:11:49.090423 2710 ts_tablet_manager.cc:1397] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:49.090471 2711 consensus_queue.cc:260] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.090550 2711 ts_tablet_manager.cc:1428] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:49.090564 2710 raft_consensus.cc:348] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.090621 2710 raft_consensus.cc:374] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:49.090639 2710 raft_consensus.cc:729] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afd132a3a074b25b54d9202e2b2dcc2, State: Initialized, Role: FOLLOWER
I20260928 23:11:49.090695 2710 consensus_queue.cc:260] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.090777 2710 ts_tablet_manager.cc:1428] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:49.090919 2712 tablet_bootstrap.cc:492] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: No bootstrap required, opened a new log
I20260928 23:11:49.090968 2712 ts_tablet_manager.cc:1397] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:49.091068 2712 raft_consensus.cc:348] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.091130 2712 raft_consensus.cc:374] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:49.091147 2712 raft_consensus.cc:729] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef118ef11934a05b27630106bf19fc4, State: Initialized, Role: FOLLOWER
I20260928 23:11:49.091192 2712 consensus_queue.cc:260] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.091274 2712 ts_tablet_manager.cc:1428] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:49.105836 2682 raft_consensus.cc:482] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:11:49.105921 2682 raft_consensus.cc:504] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.106073 2682 leader_election.cc:288] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:49.106249 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474f19214c4c4da88723697eac8f2063" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4" is_pre_election: true
I20260928 23:11:49.106247 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474f19214c4c4da88723697eac8f2063" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" is_pre_election: true
I20260928 23:11:49.106338 2552 raft_consensus.cc:2454] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:49.106343 2412 raft_consensus.cc:2454] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 0.
I20260928 23:11:49.106477 2205 leader_election.cc:302] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef118ef11934a05b27630106bf19fc4, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:49.106575 2682 raft_consensus.cc:2781] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:11:49.106612 2682 raft_consensus.cc:482] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:11:49.106626 2682 raft_consensus.cc:3037] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.107247 2682 raft_consensus.cc:504] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.107357 2682 leader_election.cc:288] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901), 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071)
I20260928 23:11:49.107475 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474f19214c4c4da88723697eac8f2063" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef118ef11934a05b27630106bf19fc4"
I20260928 23:11:49.107501 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474f19214c4c4da88723697eac8f2063" candidate_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0afd132a3a074b25b54d9202e2b2dcc2"
I20260928 23:11:49.107553 2552 raft_consensus.cc:3037] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.107574 2412 raft_consensus.cc:3037] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:49.108266 2552 raft_consensus.cc:2454] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:49.108290 2412 raft_consensus.cc:2454] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa796e3ec7d14ed3848f4178ff18a6c3 in term 1.
I20260928 23:11:49.108410 2206 leader_election.cc:302] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0afd132a3a074b25b54d9202e2b2dcc2, aa796e3ec7d14ed3848f4178ff18a6c3; no voters:
I20260928 23:11:49.108551 2682 raft_consensus.cc:2781] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:49.108595 2682 raft_consensus.cc:686] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa796e3ec7d14ed3848f4178ff18a6c3, State: Running, Role: LEADER
I20260928 23:11:49.108633 2682 consensus_queue.cc:237] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } }
I20260928 23:11:49.109138 2115 catalog_manager.cc:5495] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa796e3ec7d14ed3848f4178ff18a6c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 } health_report { overall_health: UNKNOWN } } }
I20260928 23:11:49.119902 2552 raft_consensus.cc:1261] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:49.119913 2412 raft_consensus.cc:1261] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Refusing update from remote peer aa796e3ec7d14ed3848f4178ff18a6c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:11:49.120081 2624 consensus_queue.cc:1035] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0afd132a3a074b25b54d9202e2b2dcc2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:11:49.120185 2682 consensus_queue.cc:1035] T 474f19214c4c4da88723697eac8f2063 P aa796e3ec7d14ed3848f4178ff18a6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef118ef11934a05b27630106bf19fc4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:11:49.123843 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 23:11:50.640466 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:50.640606 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:11:50.641408 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c94a067c63fd45c2a1b11f8d80c33667: Sending DeleteTablet for 3 replicas of tablet 474f19214c4c4da88723697eac8f2063
I20260928 23:11:50.641631 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 474f19214c4c4da88723697eac8f2063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42832
I20260928 23:11:50.641722 2393 tablet_service.cc:1514] Processing DeleteTablet for tablet 474f19214c4c4da88723697eac8f2063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44388
I20260928 23:11:50.641783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2188
I20260928 23:11:50.642201 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 474f19214c4c4da88723697eac8f2063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:11:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55016
I20260928 23:11:50.642383 2752 tablet_replica.cc:330] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: stopping tablet replica
I20260928 23:11:50.642434 2752 raft_consensus.cc:2229] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:50.642464 2752 raft_consensus.cc:2258] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:50.642683 2752 ts_tablet_manager.cc:1905] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:50.642786 2751 tablet_replica.cc:330] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: stopping tablet replica
I20260928 23:11:50.642868 2751 raft_consensus.cc:2229] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:11:50.642936 2751 raft_consensus.cc:2258] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:11:50.643193 2751 ts_tablet_manager.cc:1905] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:11:50.644253 2752 ts_tablet_manager.cc:1918] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:11:50.644309 2752 log.cc:1196] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/474f19214c4c4da88723697eac8f2063
I20260928 23:11:50.644503 2752 ts_tablet_manager.cc:1939] T 474f19214c4c4da88723697eac8f2063 P 2ef118ef11934a05b27630106bf19fc4: Deleting consensus metadata
I20260928 23:11:50.644728 2751 ts_tablet_manager.cc:1918] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 23:11:50.644819 2751 log.cc:1196] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/474f19214c4c4da88723697eac8f2063
I20260928 23:11:50.644860 2101 catalog_manager.cc:4841] TS 2ef118ef11934a05b27630106bf19fc4 (127.0.98.3:43071): tablet 474f19214c4c4da88723697eac8f2063 (table test-table [id=708c640fd08747baa025d831f5352d36]) successfully deleted
I20260928 23:11:50.645009 2751 ts_tablet_manager.cc:1939] T 474f19214c4c4da88723697eac8f2063 P 0afd132a3a074b25b54d9202e2b2dcc2: Deleting consensus metadata
I20260928 23:11:50.645354 2102 catalog_manager.cc:4841] TS 0afd132a3a074b25b54d9202e2b2dcc2 (127.0.98.2:42901): tablet 474f19214c4c4da88723697eac8f2063 (table test-table [id=708c640fd08747baa025d831f5352d36]) successfully deleted
W20260928 23:11:50.647543 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39029 (error 108)
I20260928 23:11:50.647960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2328
I20260928 23:11:50.648008 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:50.647844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39029 (local address 127.0.0.1:42876)
0928 23:11:50.647936 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:50.647974 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
W20260928 23:11:50.648113 2101 catalog_manager.cc:4584] TS aa796e3ec7d14ed3848f4178ff18a6c3 (127.0.98.1:39029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 474f19214c4c4da88723697eac8f2063: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39029: connect: Connection refused (error 111)
I20260928 23:11:50.652420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2468
I20260928 23:11:50.657022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2084
2026-09-28T23:11:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5711 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:50 dist-test-slave-6vgd krb5kdc[2757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:50 dist-test-slave-6vgd krb5kdc[2757](info): set up 2 sockets
Sep 28 23:11:50 dist-test-slave-6vgd krb5kdc[2757](info): commencing operation
krb5kdc: starting...
W20260928 23:11:51.600682 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:51 dist-test-slave-6vgd krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:51.707108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38171
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:51.784096 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:51.784305 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:51.784325 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:51.785485 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:51.785524 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:51.785537 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:51.785550 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:51.785573 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:51.787058 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:51.787276 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:51.787489 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:51.792130 2775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:51.792138 2778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:51.792145 2776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:51.792260 2770 server_base.cc:1034] running on GCE node
I20260928 23:11:51.792619 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:51.792903 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:51.794041 2770 hybrid_clock.cc:648] HybridClock initialized: now 1790637111794027 us; error 27 us; skew 500 ppm
Sep 28 23:11:51 dist-test-slave-6vgd krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:51.796344 2770 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:51.796753 2770 webserver.cc:460] Webserver started at http://127.0.98.62:43313/ using document root <none> and password file <none>
I20260928 23:11:51.796943 2770 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:51.796990 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:51.797109 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:51.798142 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "051da9d5e3b24935341e5914e12c4ac4"
server_key_iv: "cd7495f0eb94456ea069094d96a1aebf"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.798472 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "051da9d5e3b24935341e5914e12c4ac4"
server_key_iv: "cd7495f0eb94456ea069094d96a1aebf"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.799706 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:11:51.800377 2785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:51.800580 2770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:51.800666 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "051da9d5e3b24935341e5914e12c4ac4"
server_key_iv: "cd7495f0eb94456ea069094d96a1aebf"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.800732 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:51.815428 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:51.817260 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:51.817392 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:51.821028 2770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38171
I20260928 23:11:51.821066 2837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38171 every 8 connection(s)
I20260928 23:11:51.821540 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:51.822126 2838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:51.824278 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d: Bootstrap starting.
I20260928 23:11:51.824817 2838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:51.825070 2838 log.cc:824] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:51.825662 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d: No bootstrap required, opened a new log
I20260928 23:11:51.827191 2838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } }
I20260928 23:11:51.827325 2838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:51.827346 2838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4c45aa13aa4635ab7e73f1f6089d5d, State: Initialized, Role: FOLLOWER
I20260928 23:11:51.827476 2838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } }
I20260928 23:11:51.827538 2838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:51.827559 2838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:51.827601 2838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:51.828419 2838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } }
I20260928 23:11:51.828500 2838 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4c45aa13aa4635ab7e73f1f6089d5d; no voters:
I20260928 23:11:51.828640 2838 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:51.828684 2841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:51.828862 2841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [term 1 LEADER]: Becoming Leader. State: Replica: 1f4c45aa13aa4635ab7e73f1f6089d5d, State: Running, Role: LEADER
I20260928 23:11:51.828872 2838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:51.828956 2841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } }
I20260928 23:11:51.829295 2842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } } }
I20260928 23:11:51.829303 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4c45aa13aa4635ab7e73f1f6089d5d. Latest consensus state: current_term: 1 leader_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4c45aa13aa4635ab7e73f1f6089d5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38171 } } }
I20260928 23:11:51.829455 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:51.829680 2850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:51.829448 2842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:51.830160 2850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:51.830929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2770
I20260928 23:11:51.831019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:51.831176 392 external_mini_cluster.cc:1428] Setting key 2f3783ffc998631f1e34733ecb0660ee
I20260928 23:11:51.831683 2850 catalog_manager.cc:1347] Generated new cluster ID: e1bd9b9aa6d54ec4af1cc25f7084e5a0
I20260928 23:11:51.831743 2850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:11:51 dist-test-slave-6vgd krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:51.842468 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:51.831700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.98.62:38171)
0928 23:11:51.831968 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:51.831978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:51.832006 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:11:51.832113 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:51.832120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:51.832171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:51.832305 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:51.832967 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:51.834425 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:51.835429 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:51.836381 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:51.836488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:51.839106 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:51.839110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:51.839110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:51.839122 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:51.840951 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:51.841348 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:51.841350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:51.841350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:51.841449 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:51.841625 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:51.841625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:51.841626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:51.841899 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:51.842015 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:51.842121 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 23:11:51.842243 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
I20260928 23:11:51.853688 2850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:51.854413 2850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:51.861322 2850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4c45aa13aa4635ab7e73f1f6089d5d: Generated new TSK 0
I20260928 23:11:51.861500 2850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:51.889972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:51.968876 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:51.969100 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:51.969129 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:51.970304 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:51.970348 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:51.970381 2864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:51.971872 2864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:51.972101 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:51.972314 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:51.977041 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:51.977031 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:51.977099 2864 server_base.cc:1034] running on GCE node
W20260928 23:11:51.977035 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:51.977485 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:51.977723 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:51.978866 2864 hybrid_clock.cc:648] HybridClock initialized: now 1790637111978851 us; error 28 us; skew 500 ppm
Sep 28 23:11:51 dist-test-slave-6vgd krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:51.981184 2864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:51.981628 2864 webserver.cc:460] Webserver started at http://127.0.98.1:34887/ using document root <none> and password file <none>
I20260928 23:11:51.981833 2864 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:51.981879 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:51.982010 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:51.983035 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "a913861c787f40d5b49c40c3e4fdfa35"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "657549dc2f37345ae9424d74a0fe1373"
server_key_iv: "db0b9b3c6090111f70cc84febe23fe97"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.983350 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a913861c787f40d5b49c40c3e4fdfa35"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "657549dc2f37345ae9424d74a0fe1373"
server_key_iv: "db0b9b3c6090111f70cc84febe23fe97"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.984555 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:51.985294 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:51.985455 2864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:51.985533 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "a913861c787f40d5b49c40c3e4fdfa35"
format_stamp: "Formatted at 2026-09-28 23:11:51 on dist-test-slave-6vgd"
server_key: "657549dc2f37345ae9424d74a0fe1373"
server_key_iv: "db0b9b3c6090111f70cc84febe23fe97"
server_key_version: "encryptionkey@0"
I20260928 23:11:51.985599 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:52.025766 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:52.027736 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:52.027871 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:52.028093 2864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:52.028386 2864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:52.028421 2864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.028451 2864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:52.028468 2864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.034025 2864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38929
I20260928 23:11:52.034112 2992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38929 every 8 connection(s)
I20260928 23:11:52.034382 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:11:52.034945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2864
I20260928 23:11:52.035027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:52.035178 392 external_mini_cluster.cc:1428] Setting key 4f5f63f6051d1e70c368675e8ad43959
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:11:52.043581 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.035041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49037 (local address 127.0.98.62:38171)
0928 23:11:52.035253 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:52.035256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:52.035324 (+ 68us) server_negotiation.cc:407] Connection header received
0928 23:11:52.036343 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:52.036344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:52.036359 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:52.036408 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:52.037018 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.037579 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.038534 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.038792 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.038857 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.040866 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:52.040868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:52.040868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:52.040875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:52.042381 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.042728 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.042729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.042729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.042791 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.042997 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.042997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.042998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.043128 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:52.043189 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:52.043407 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 23:11:52.043482 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260928 23:11:52.043701 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.035030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38171 (local address 127.0.98.1:49037)
0928 23:11:52.035200 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:52.035243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:52.036242 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:52.036487 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:52.036491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:52.036734 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:52.036932 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.036937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.037651 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.037652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.038459 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.038461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.038871 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.038872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.038995 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.039452 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:52.039460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:52.040772 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:52.042453 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.042458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.042460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.042661 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.042855 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.042856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.042856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.042943 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.043187 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:52.043190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:52.043297 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 23:11:52.043372 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:11:52.043484 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 23:11:52.044189 2993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38171
I20260928 23:11:52.044265 2993 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:52.044420 2993 heartbeater.cc:511] Master 127.0.98.62:38171 requested a full tablet report, sending...
I20260928 23:11:52.044872 2802 ts_manager.cc:194] Registered new tserver with Master: a913861c787f40d5b49c40c3e4fdfa35 (127.0.98.1:38929)
I20260928 23:11:52.045547 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:52.058301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:52.137795 2998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:52.138007 2998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:52.138032 2998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:52.139222 2998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:52.139261 2998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:52.139288 2998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:52.140754 2998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:52.140946 2998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:52.141132 2998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:52.146625 3004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:52.146617 3003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:52.146957 2998 server_base.cc:1034] running on GCE node
W20260928 23:11:52.146771 3006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:52.147320 2998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:52.147598 2998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:52.148758 2998 hybrid_clock.cc:648] HybridClock initialized: now 1790637112148739 us; error 33 us; skew 500 ppm
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:52.151422 2998 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:52.151834 2998 webserver.cc:460] Webserver started at http://127.0.98.2:33657/ using document root <none> and password file <none>
I20260928 23:11:52.152052 2998 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:52.152097 2998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:52.152236 2998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:52.153318 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "e0b17fb5e57a456585e1faac5c5403b6"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "a918277f987d133c621dd0b3f2f88f23"
server_key_iv: "a7a552fa0b9575939a83262fcad8cad1"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.153640 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e0b17fb5e57a456585e1faac5c5403b6"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "a918277f987d133c621dd0b3f2f88f23"
server_key_iv: "a7a552fa0b9575939a83262fcad8cad1"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.154838 2998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:52.155503 3013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.155659 2998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:52.155740 2998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "e0b17fb5e57a456585e1faac5c5403b6"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "a918277f987d133c621dd0b3f2f88f23"
server_key_iv: "a7a552fa0b9575939a83262fcad8cad1"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.155803 2998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:52.162689 2998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:52.164297 2998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:52.164420 2998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:52.164602 2998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:52.164857 2998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:52.164889 2998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.164914 2998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:52.164947 2998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.170363 2998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45355
I20260928 23:11:52.170413 3126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45355 every 8 connection(s)
I20260928 23:11:52.170688 2998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:11:52.172564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 2998
I20260928 23:11:52.172662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:52.172840 392 external_mini_cluster.cc:1428] Setting key 83320d55b25739164837fa99d8d2a509
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:52.180124 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.171321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35525 (local address 127.0.98.62:38171)
0928 23:11:52.171396 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:52.171399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:52.171535 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:11:52.172580 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:52.172581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:52.172590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:52.172657 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:52.173435 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.173961 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.175194 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.175407 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.175463 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.177551 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:52.177552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:52.177553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:52.177561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:52.178931 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.179271 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.179272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.179273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.179338 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.179543 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.179544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.179545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.179669 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:52.179740 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:52.179945 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 23:11:52.180034 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:11:52.180183 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.171282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38171 (local address 127.0.98.2:35525)
0928 23:11:52.171419 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:52.171453 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:52.172475 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:52.172737 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:52.172741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:52.173078 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:52.173349 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.173356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.174041 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.174042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.175117 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.175119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.175472 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.175474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.175597 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.176057 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:52.176065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:52.177466 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:52.179001 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.179006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.179007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.179205 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.179398 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.179399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.179399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.179492 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.179727 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:52.179730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:52.179835 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:11:52.179908 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:11:52.180025 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:11:52.180606 3127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38171
I20260928 23:11:52.180691 3127 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:52.180830 3127 heartbeater.cc:511] Master 127.0.98.62:38171 requested a full tablet report, sending...
I20260928 23:11:52.181159 2802 ts_manager.cc:194] Registered new tserver with Master: e0b17fb5e57a456585e1faac5c5403b6 (127.0.98.2:45355)
I20260928 23:11:52.181555 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:52.196103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:52.272871 3132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:52.273089 3132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:52.273120 3132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:52.274300 3132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:52.274346 3132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:52.274382 3132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:52.275861 3132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:52.276103 3132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:52.276314 3132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:52.281176 3138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:52.281180 3137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:52.281287 3132 server_base.cc:1034] running on GCE node
W20260928 23:11:52.281246 3140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:52.281596 3132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:52.281819 3132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:52.282958 3132 hybrid_clock.cc:648] HybridClock initialized: now 1790637112282949 us; error 32 us; skew 500 ppm
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:52.285315 3132 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:52.285748 3132 webserver.cc:460] Webserver started at http://127.0.98.3:44035/ using document root <none> and password file <none>
I20260928 23:11:52.285955 3132 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:52.286003 3132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:52.286126 3132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:52.287130 3132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "f267deb2bfbd4697b7585d9aebfcc453"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "69ebc46357420f0763b8fe7d802111c5"
server_key_iv: "4efdf630a1c807fa30166aea9f5e3da3"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.287451 3132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f267deb2bfbd4697b7585d9aebfcc453"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "69ebc46357420f0763b8fe7d802111c5"
server_key_iv: "4efdf630a1c807fa30166aea9f5e3da3"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.288687 3132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:52.289377 3147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.289530 3132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:52.289620 3132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "f267deb2bfbd4697b7585d9aebfcc453"
format_stamp: "Formatted at 2026-09-28 23:11:52 on dist-test-slave-6vgd"
server_key: "69ebc46357420f0763b8fe7d802111c5"
server_key_iv: "4efdf630a1c807fa30166aea9f5e3da3"
server_key_version: "encryptionkey@0"
I20260928 23:11:52.289676 3132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:52.310319 3132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:52.312057 3132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:52.312165 3132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:52.312350 3132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:52.312600 3132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:52.312629 3132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.312649 3132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:52.312662 3132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:52.317843 3132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35159
I20260928 23:11:52.317895 3260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35159 every 8 connection(s)
I20260928 23:11:52.318174 3132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:11:52.320015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3132
I20260928 23:11:52.320114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:52.320353 392 external_mini_cluster.cc:1428] Setting key 43c1ee497d68252d4992d457aa0b3bef
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:52.326797 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.318808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43127 (local address 127.0.98.62:38171)
0928 23:11:52.318868 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:52.318870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:52.319096 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:11:52.319989 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:52.319990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:52.320003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:52.320073 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:52.320811 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.321466 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.322407 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.322586 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.322635 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.324420 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:52.324421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:52.324422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:52.324428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:52.325665 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.325996 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.325997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.325998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.326054 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:52.326259 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:52.326260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:52.326260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:52.326374 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:52.326429 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:52.326666 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 23:11:52.326733 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:11:52.326892 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:52.318788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38171 (local address 127.0.98.3:43127)
0928 23:11:52.318972 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:52.319006 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:52.319891 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:52.320216 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:52.320220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:52.320536 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:52.320744 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.320749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.321545 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.321546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.322341 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:52.322343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:52.322642 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:52.322642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:52.322732 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:52.323127 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:52.323134 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:52.324347 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:52.325720 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.325723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.325725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.325916 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.326110 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:52.326110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:52.326111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:52.326195 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:52.326423 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:52.326426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:52.326540 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:11:52.326616 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:11:52.326729 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260928 23:11:52.327293 3261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38171
I20260928 23:11:52.327376 3261 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:52.327543 3261 heartbeater.cc:511] Master 127.0.98.62:38171 requested a full tablet report, sending...
I20260928 23:11:52.327913 2802 ts_manager.cc:194] Registered new tserver with Master: f267deb2bfbd4697b7585d9aebfcc453 (127.0.98.3:35159)
I20260928 23:11:52.328254 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43127
I20260928 23:11:52.331892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:11:52.341750 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:11:52.340873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.0.98.62:38171)
0928 23:11:52.340922 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:52.340924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:52.341019 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:11:52.341085 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:52.341086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:52.341096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:52.341141 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:52.341678 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41756: BlockingRecv error: recv got EOF from 127.0.0.1:41756 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:11:52.342610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2864
I20260928 23:11:52.346225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2998
I20260928 23:11:52.349707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3132
I20260928 23:11:52.353142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 2770
2026-09-28T23:11:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1695 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[3273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[3273](info): set up 2 sockets
Sep 28 23:11:52 dist-test-slave-6vgd krb5kdc[3273](info): commencing operation
krb5kdc: starting...
W20260928 23:11:53.335181 392 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 23:11:53.486828 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.152s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 23:11:53.689021 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.106s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/test-user.keytab.
W20260928 23:11:54.353300 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.664s user 0.001s sys 0.000s
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260928 23:11:54.556166 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.183s user 0.000s sys 0.002s
I20260928 23:11:54.556486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46159
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:54.633006 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:54.633203 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:54.633253 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:54.634382 3286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:54.634416 3286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:54.634429 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:54.634441 3286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:54.634454 3286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:54.635864 3286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:54.636045 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:54.636206 3286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:54.640463 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:54.640476 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:54.640507 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:54.640643 3286 server_base.cc:1034] running on GCE node
I20260928 23:11:54.640851 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:54.641119 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:54.642257 3286 hybrid_clock.cc:648] HybridClock initialized: now 1790637114642238 us; error 33 us; skew 500 ppm
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:54.647332 3286 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:54.647749 3286 webserver.cc:460] Webserver started at http://127.0.98.62:32895/ using document root <none> and password file <none>
I20260928 23:11:54.647949 3286 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:54.647995 3286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:54.648116 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:54.649129 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "83a5026c9cad4554abb7b86a67626aa1"
format_stamp: "Formatted at 2026-09-28 23:11:54 on dist-test-slave-6vgd"
server_key: "e824f22030635d6dd88dbc90b54fadad"
server_key_iv: "530d19582aad162e2458db0a85ca52f9"
server_key_version: "encryptionkey@0"
I20260928 23:11:54.649539 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "83a5026c9cad4554abb7b86a67626aa1"
format_stamp: "Formatted at 2026-09-28 23:11:54 on dist-test-slave-6vgd"
server_key: "e824f22030635d6dd88dbc90b54fadad"
server_key_iv: "530d19582aad162e2458db0a85ca52f9"
server_key_version: "encryptionkey@0"
I20260928 23:11:54.650743 3286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:54.651392 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:54.651536 3286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:54.651615 3286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "83a5026c9cad4554abb7b86a67626aa1"
format_stamp: "Formatted at 2026-09-28 23:11:54 on dist-test-slave-6vgd"
server_key: "e824f22030635d6dd88dbc90b54fadad"
server_key_iv: "530d19582aad162e2458db0a85ca52f9"
server_key_version: "encryptionkey@0"
I20260928 23:11:54.651671 3286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:54.660386 3286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:54.662319 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:54.662446 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:54.666002 3286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46159
I20260928 23:11:54.666043 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46159 every 8 connection(s)
I20260928 23:11:54.666456 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:54.667325 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:54.669514 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1: Bootstrap starting.
I20260928 23:11:54.670043 3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:54.670289 3354 log.cc:824] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:54.670496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3286
I20260928 23:11:54.670581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:54.670753 392 external_mini_cluster.cc:1428] Setting key c20ed80a1a497747f2a796ba9f658787
I20260928 23:11:54.670982 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1: No bootstrap required, opened a new log
I20260928 23:11:54.673448 3354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } }
I20260928 23:11:54.673575 3354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:54.673596 3354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a5026c9cad4554abb7b86a67626aa1, State: Initialized, Role: FOLLOWER
I20260928 23:11:54.673699 3354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } }
I20260928 23:11:54.673771 3354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:54.673794 3354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:54.673839 3354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:54.674367 3354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } }
I20260928 23:11:54.674474 3354 leader_election.cc:302] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a5026c9cad4554abb7b86a67626aa1; no voters:
I20260928 23:11:54.674619 3354 leader_election.cc:288] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:54.674645 3359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:54.674801 3359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 83a5026c9cad4554abb7b86a67626aa1, State: Running, Role: LEADER
I20260928 23:11:54.674862 3354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:54.674921 3359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } }
I20260928 23:11:54.675751 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a5026c9cad4554abb7b86a67626aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } } }
I20260928 23:11:54.675858 3360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a5026c9cad4554abb7b86a67626aa1. Latest consensus state: current_term: 1 leader_uuid: "83a5026c9cad4554abb7b86a67626aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5026c9cad4554abb7b86a67626aa1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46159 } } }
I20260928 23:11:54.675904 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:54.675930 3360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:54.676185 3372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:54.676796 3372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:54.678272 3372 catalog_manager.cc:1347] Generated new cluster ID: 72071be40aa94fe5aabb6ceb7aa04712
I20260928 23:11:54.678316 3372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:11:54.681641 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:54.671197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49396 (local address 127.0.98.62:46159)
0928 23:11:54.671370 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:54.671378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:54.671402 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:11:54.672139 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:54.672145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:54.672205 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:54.672352 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:54.673052 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:54.674102 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:54.675100 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:54.675707 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:54.675807 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:54.678318 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:54.678322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:54.678323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:54.678333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:54.680121 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:54.680472 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:54.680474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:54.680475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:54.680590 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:54.680761 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:54.680762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:54.680762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:54.681052 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:54.681201 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:54.681300 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:11:54.681429 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 23:11:54.682669 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:11:54.688472 3372 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:54.689121 3372 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:54.696413 3372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a5026c9cad4554abb7b86a67626aa1: Generated new TSK 0
I20260928 23:11:54.696566 3372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:11:54.697367 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:54.693599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.98.62:46159)
0928 23:11:54.693635 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:54.693636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:54.693727 (+ 91us) server_negotiation.cc:407] Connection header received
0928 23:11:54.693770 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:54.693771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:54.693780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:54.693847 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:54.694166 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:54.694953 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:54.695945 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:54.696270 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:54.696330 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:54.696988 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:54.696989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:54.696989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:11:54.697136 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:54.697151 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:11:54.697156 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:54.697204 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:54.697295 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:11:54.697318 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":17}
I20260928 23:11:54.699307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3286
2026-09-28T23:11:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2344 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3388](info): set up 2 sockets
Sep 28 23:11:54 dist-test-slave-6vgd krb5kdc[3388](info): commencing operation
krb5kdc: starting...
W20260928 23:11:55.640761 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:55 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:55.753974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:55.833763 3401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:55.833988 3401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:55.834018 3401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:55.835208 3401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:55.835258 3401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:55.835274 3401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:55.835290 3401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:55.835305 3401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:55.836814 3401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:55.837070 3401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:55.837324 3401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:55.842096 3409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:55.842263 3407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:55.842319 3406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:55.842343 3401 server_base.cc:1034] running on GCE node
I20260928 23:11:55.842561 3401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:55.842818 3401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:55.843982 3401 hybrid_clock.cc:648] HybridClock initialized: now 1790637115843965 us; error 26 us; skew 500 ppm
Sep 28 23:11:55 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:55.846472 3401 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:55.846913 3401 webserver.cc:460] Webserver started at http://127.0.98.62:36279/ using document root <none> and password file <none>
I20260928 23:11:55.847128 3401 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:55.847177 3401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:55.847309 3401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:55.848361 3401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "80d1e6810d1d4d1d96acd77802bb8ca3"
format_stamp: "Formatted at 2026-09-28 23:11:55 on dist-test-slave-6vgd"
server_key: "d1cc3f69ff8d9deb357103bc2901f2cd"
server_key_iv: "5489926c0fe5172cafeb70077bc3299b"
server_key_version: "encryptionkey@0"
I20260928 23:11:55.848683 3401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "80d1e6810d1d4d1d96acd77802bb8ca3"
format_stamp: "Formatted at 2026-09-28 23:11:55 on dist-test-slave-6vgd"
server_key: "d1cc3f69ff8d9deb357103bc2901f2cd"
server_key_iv: "5489926c0fe5172cafeb70077bc3299b"
server_key_version: "encryptionkey@0"
I20260928 23:11:55.849956 3401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:55.850662 3416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:55.850814 3401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:55.850894 3401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "80d1e6810d1d4d1d96acd77802bb8ca3"
format_stamp: "Formatted at 2026-09-28 23:11:55 on dist-test-slave-6vgd"
server_key: "d1cc3f69ff8d9deb357103bc2901f2cd"
server_key_iv: "5489926c0fe5172cafeb70077bc3299b"
server_key_version: "encryptionkey@0"
I20260928 23:11:55.850955 3401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:55.858167 3401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:55.860190 3401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:55.860329 3401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:55.864019 3401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41765
I20260928 23:11:55.864053 3468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41765 every 8 connection(s)
I20260928 23:11:55.864372 3401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:55.865324 3469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:55.867550 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3: Bootstrap starting.
I20260928 23:11:55.868095 3469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:55.868346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3401
I20260928 23:11:55.868373 3469 log.cc:824] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:55.868446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:55.868623 392 external_mini_cluster.cc:1428] Setting key fbe61543d5a7b7c11f5b2996032bd8e7
I20260928 23:11:55.869189 3469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3: No bootstrap required, opened a new log
I20260928 23:11:55.871841 3469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } }
I20260928 23:11:55.872021 3469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:55.872058 3469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d1e6810d1d4d1d96acd77802bb8ca3, State: Initialized, Role: FOLLOWER
I20260928 23:11:55.872233 3469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } }
I20260928 23:11:55.872318 3469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:55.872349 3469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:55.872397 3469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:55.872990 3469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } }
I20260928 23:11:55.873101 3469 leader_election.cc:302] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80d1e6810d1d4d1d96acd77802bb8ca3; no voters:
I20260928 23:11:55.873294 3469 leader_election.cc:288] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:55.873322 3474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:55.873492 3474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [term 1 LEADER]: Becoming Leader. State: Replica: 80d1e6810d1d4d1d96acd77802bb8ca3, State: Running, Role: LEADER
I20260928 23:11:55.873545 3469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:55.873622 3474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } }
I20260928 23:11:55.874056 3475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } } }
I20260928 23:11:55.874138 3475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:55.874338 3476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80d1e6810d1d4d1d96acd77802bb8ca3. Latest consensus state: current_term: 1 leader_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80d1e6810d1d4d1d96acd77802bb8ca3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41765 } } }
I20260928 23:11:55.874470 3476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:55.874924 3483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:55.875435 3483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:11:55 dist-test-slave-6vgd krb5kdc[3388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:55.877368 3483 catalog_manager.cc:1347] Generated new cluster ID: 5865f3c394254878a6cfea173dcc5451
I20260928 23:11:55.877408 3483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:11:55.892316 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:55.869115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.0.98.62:41765)
0928 23:11:55.869309 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:55.869319 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:55.869354 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:11:55.869853 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:55.869862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:55.869929 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:55.870090 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:55.870654 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:55.872013 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:55.872886 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:55.873529 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:55.873717 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:55.876454 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:55.876459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:55.876460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:55.876472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:55.890396 (+ 13924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:55.890956 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:55.890960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:55.890960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:55.891084 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:55.891289 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:55.891290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:55.891290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:55.891628 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:55.891773 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:55.891906 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:11:55.892056 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":128,"threads_started":1}
I20260928 23:11:55.895961 3483 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:55.896528 3483 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:55.900161 3483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80d1e6810d1d4d1d96acd77802bb8ca3: Generated new TSK 0
I20260928 23:11:55.900334 3483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:55.928146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:56.006594 3495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:56.006796 3495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:56.006817 3495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:56.007952 3495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:56.007995 3495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:56.008085 3495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:56.009730 3495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:56.009967 3495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:56.010167 3495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:56.014827 3503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.014806 3501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.014868 3500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:56.014978 3495 server_base.cc:1034] running on GCE node
I20260928 23:11:56.015218 3495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:56.015440 3495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:56.016577 3495 hybrid_clock.cc:648] HybridClock initialized: now 1790637116016567 us; error 28 us; skew 500 ppm
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:56.018816 3495 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:56.019201 3495 webserver.cc:460] Webserver started at http://127.0.98.1:38897/ using document root <none> and password file <none>
I20260928 23:11:56.019382 3495 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:56.019428 3495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:56.019547 3495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:56.020525 3495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "43ec792afe634bf08923bf6490de3889"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "83ff89b29107125f9f0221da882efd4c"
server_key_iv: "948fe283c5e833e8309ed395b98fc698"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.020840 3495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "43ec792afe634bf08923bf6490de3889"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "83ff89b29107125f9f0221da882efd4c"
server_key_iv: "948fe283c5e833e8309ed395b98fc698"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.022054 3495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:56.022734 3510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.022897 3495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:56.022994 3495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "43ec792afe634bf08923bf6490de3889"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "83ff89b29107125f9f0221da882efd4c"
server_key_iv: "948fe283c5e833e8309ed395b98fc698"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.023053 3495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:56.035000 3495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:56.036651 3495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:56.036775 3495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:56.036984 3495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:56.037283 3495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:56.037321 3495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.037349 3495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:56.037374 3495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.042833 3495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34753
I20260928 23:11:56.042882 3623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34753 every 8 connection(s)
I20260928 23:11:56.043174 3495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:56.051427 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.043851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55613 (local address 127.0.98.62:41765)
0928 23:11:56.043911 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:56.043913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:56.044074 (+ 161us) server_negotiation.cc:407] Connection header received
0928 23:11:56.044560 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:56.044561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:56.044573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:56.044624 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:56.045555 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.046054 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.047001 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.047262 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.047317 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.049120 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:56.049121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:56.049121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:56.049129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:56.050198 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.050568 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.050569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.050569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.050625 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.050829 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.050830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.050830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.050936 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:56.050989 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:56.051227 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 23:11:56.051335 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:11:56.051542 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.043798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41765 (local address 127.0.98.1:55613)
0928 23:11:56.043949 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:56.043993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:56.044469 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:56.044691 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:56.044695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:56.045279 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:56.045468 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.045473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.046131 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.046132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.046923 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.046925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.047331 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.047332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.047442 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.047820 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:56.047827 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:56.049037 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:56.050299 (+ 1262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.050306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.050308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.050504 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.050677 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.050677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.050678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.050775 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.050987 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:56.050992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:56.051117 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 23:11:56.051200 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:11:56.051344 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 23:11:56.051961 3624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41765
I20260928 23:11:56.052037 3624 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:56.052165 3624 heartbeater.cc:511] Master 127.0.98.62:41765 requested a full tablet report, sending...
I20260928 23:11:56.052627 3433 ts_manager.cc:194] Registered new tserver with Master: 43ec792afe634bf08923bf6490de3889 (127.0.98.1:34753)
I20260928 23:11:56.052928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3495
I20260928 23:11:56.053020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:56.053196 392 external_mini_cluster.cc:1428] Setting key a9d5a398bb2d3875b5280bf0a204d766
I20260928 23:11:56.053447 3433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55613
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:56.075625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:56.155017 3629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:56.155220 3629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:56.155243 3629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:56.156383 3629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:56.156416 3629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:56.156443 3629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:56.157864 3629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:56.158062 3629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:56.158239 3629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:56.162813 3637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.162796 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.162901 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:56.162978 3629 server_base.cc:1034] running on GCE node
I20260928 23:11:56.163255 3629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:56.163486 3629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:56.164621 3629 hybrid_clock.cc:648] HybridClock initialized: now 1790637116164605 us; error 33 us; skew 500 ppm
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:56.166905 3629 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:56.167301 3629 webserver.cc:460] Webserver started at http://127.0.98.2:39089/ using document root <none> and password file <none>
I20260928 23:11:56.167495 3629 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:56.167542 3629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:56.167662 3629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:56.168670 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "5a6be28e160f4ae0b31b41d71b0b8fff"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "a25385ebcf0dee4fa412bacc1fdd66c5"
server_key_iv: "d2de2bd8deaf0437974f33b08422124c"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.169003 3629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5a6be28e160f4ae0b31b41d71b0b8fff"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "a25385ebcf0dee4fa412bacc1fdd66c5"
server_key_iv: "d2de2bd8deaf0437974f33b08422124c"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.170284 3629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 23:11:56.170929 3644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.171073 3629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:56.171149 3629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "5a6be28e160f4ae0b31b41d71b0b8fff"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "a25385ebcf0dee4fa412bacc1fdd66c5"
server_key_iv: "d2de2bd8deaf0437974f33b08422124c"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.171213 3629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:56.190711 3629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:56.192551 3629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:56.192691 3629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:56.192912 3629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:56.193202 3629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:56.193259 3629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.193289 3629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:56.193311 3629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.198714 3629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32787
I20260928 23:11:56.198765 3757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32787 every 8 connection(s)
I20260928 23:11:56.199059 3629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:11:56.199894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3629
I20260928 23:11:56.200004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:56.200174 392 external_mini_cluster.cc:1428] Setting key 8879afc1e527c4658e3890e635f74cef
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:11:56.207674 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.199715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59331 (local address 127.0.98.62:41765)
0928 23:11:56.199774 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:56.199776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:56.199964 (+ 188us) server_negotiation.cc:407] Connection header received
0928 23:11:56.200551 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:56.200552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:56.200563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:56.200608 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:56.201606 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.202101 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.203067 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.203253 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.203309 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.205273 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:56.205274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:56.205274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:56.205282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:56.206520 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.206875 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.206876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.206877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.206938 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.207160 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.207160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.207161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.207269 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:56.207320 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:56.207544 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 23:11:56.207608 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:11:56.207840 3760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.199698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41765 (local address 127.0.98.2:59331)
0928 23:11:56.199845 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:56.199880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:56.200458 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:56.200681 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:56.200685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:56.201314 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:56.201515 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.201521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.202180 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.202181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.202985 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.202987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.203320 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.203321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.203449 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.203908 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:56.203916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:56.205173 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:56.206594 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.206600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.206602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.206808 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.207006 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.207007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.207007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.207103 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.207317 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:56.207320 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:56.207445 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 23:11:56.207523 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:11:56.207640 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260928 23:11:56.208273 3758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41765
I20260928 23:11:56.208354 3758 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:56.208505 3758 heartbeater.cc:511] Master 127.0.98.62:41765 requested a full tablet report, sending...
I20260928 23:11:56.208858 3432 ts_manager.cc:194] Registered new tserver with Master: 5a6be28e160f4ae0b31b41d71b0b8fff (127.0.98.2:32787)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 23:11:56.209352 3432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:56.222056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:56.298954 3763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:56.299149 3763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:56.299170 3763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:56.300303 3763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:56.300343 3763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:56.300369 3763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:56.301841 3763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:56.302062 3763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:56.302260 3763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:56.306905 3769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.306885 3771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:56.307001 3768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:56.307036 3763 server_base.cc:1034] running on GCE node
I20260928 23:11:56.307318 3763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:56.307551 3763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:56.308686 3763 hybrid_clock.cc:648] HybridClock initialized: now 1790637116308672 us; error 28 us; skew 500 ppm
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:56.310999 3763 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:56.311399 3763 webserver.cc:460] Webserver started at http://127.0.98.3:42203/ using document root <none> and password file <none>
I20260928 23:11:56.311594 3763 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:56.311635 3763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:56.311717 3763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:56.312697 3763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "0aa0bea2f6c74e71aa7d31fd5ce31209"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "29c5ec43625a9c817e2b83158cfcc684"
server_key_iv: "78dc8093e5d1f63b4343c445168255cf"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.312992 3763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0aa0bea2f6c74e71aa7d31fd5ce31209"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "29c5ec43625a9c817e2b83158cfcc684"
server_key_iv: "78dc8093e5d1f63b4343c445168255cf"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.314191 3763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:56.314839 3778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.314980 3763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:56.315045 3763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "0aa0bea2f6c74e71aa7d31fd5ce31209"
format_stamp: "Formatted at 2026-09-28 23:11:56 on dist-test-slave-6vgd"
server_key: "29c5ec43625a9c817e2b83158cfcc684"
server_key_iv: "78dc8093e5d1f63b4343c445168255cf"
server_key_version: "encryptionkey@0"
I20260928 23:11:56.315115 3763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:56.338609 3763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:56.340488 3763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:56.340611 3763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:56.340854 3763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:56.341154 3763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:56.341187 3763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.341245 3763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:56.341272 3763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:56.346596 3763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46681
I20260928 23:11:56.346650 3891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46681 every 8 connection(s)
I20260928 23:11:56.346956 3763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:56.355253 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.347590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42807 (local address 127.0.98.62:41765)
0928 23:11:56.347663 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:56.347665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:56.347880 (+ 215us) server_negotiation.cc:407] Connection header received
0928 23:11:56.348357 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:56.348358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:56.348368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:56.348416 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:56.349373 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.349877 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.350834 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.351030 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.351092 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.352986 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:56.352987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:56.352987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:56.352993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:56.354221 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.354559 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.354559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.354560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.354603 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.354795 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.354795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.354795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.354890 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:56.354936 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:56.355136 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 23:11:56.355190 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:11:56.355441 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.347573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41765 (local address 127.0.98.3:42807)
0928 23:11:56.347759 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:56.347798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:56.348284 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:56.348484 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:56.348487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:56.349042 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:56.349281 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.349288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.349937 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.349938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.350746 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:56.350748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.351099 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.351100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:56.351193 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.351627 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:56.351635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:56.352909 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:56.354276 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.354282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.354284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.354505 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.354644 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:56.354644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:56.354645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:56.354748 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:56.354932 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:56.354935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:56.355046 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 23:11:56.355120 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:11:56.355234 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260928 23:11:56.355886 3892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41765
I20260928 23:11:56.355973 3892 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:56.356140 3892 heartbeater.cc:511] Master 127.0.98.62:41765 requested a full tablet report, sending...
I20260928 23:11:56.356228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3763
I20260928 23:11:56.356328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:56.356503 392 external_mini_cluster.cc:1428] Setting key 03efc6694870b6ab5401a93fa6d6ecae
I20260928 23:11:56.356523 3432 ts_manager.cc:194] Registered new tserver with Master: 0aa0bea2f6c74e71aa7d31fd5ce31209 (127.0.98.3:46681)
I20260928 23:11:56.356978 3432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42807
I20260928 23:11:56.357301 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637116, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:56.377753 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:56.371332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.98.62:41765)
0928 23:11:56.371403 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:56.371404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:56.371470 (+ 66us) server_negotiation.cc:407] Connection header received
0928 23:11:56.371524 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:56.371525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:56.371533 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:56.371579 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:56.372063 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.372570 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.373276 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:56.373470 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:56.373530 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:56.375736 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:56.375737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:56.375737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:56.375744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:56.376748 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.377036 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.377037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.377038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.377090 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:56.377270 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:56.377271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:56.377271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:56.377405 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:56.377500 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:56.377588 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:11:56.377662 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 23:11:56.377946 3433 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51008
I20260928 23:11:56.378297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3495
I20260928 23:11:56.382407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3629
I20260928 23:11:56.386538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3763
I20260928 23:11:56.390163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3401
2026-09-28T23:11:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1692 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3904](info): set up 2 sockets
Sep 28 23:11:56 dist-test-slave-6vgd krb5kdc[3904](info): commencing operation
krb5kdc: starting...
W20260928 23:11:57.340801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:57.446142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33369
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:57.532823 3917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:57.533027 3917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:57.533047 3917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:57.534274 3917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:57.534318 3917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:57.534332 3917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:57.534354 3917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:57.534376 3917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:57.535801 3917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:57.536029 3917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:57.536234 3917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:57.540642 3923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.540670 3922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.540668 3925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:57.540815 3917 server_base.cc:1034] running on GCE node
I20260928 23:11:57.541035 3917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:57.541342 3917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:57.542497 3917 hybrid_clock.cc:648] HybridClock initialized: now 1790637117542477 us; error 33 us; skew 500 ppm
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:57.544898 3917 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:57.545336 3917 webserver.cc:460] Webserver started at http://127.0.98.62:35913/ using document root <none> and password file <none>
I20260928 23:11:57.545543 3917 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:57.545589 3917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:57.545711 3917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:57.546788 3917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "f03e6877589647cbbf586c574a7d3375"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "98a5fa16d3d31dc70597b8c6db13ce5d"
server_key_iv: "5ea0d6a253cb5ff093572d0cbb3f5962"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.547145 3917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "f03e6877589647cbbf586c574a7d3375"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "98a5fa16d3d31dc70597b8c6db13ce5d"
server_key_iv: "5ea0d6a253cb5ff093572d0cbb3f5962"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.548337 3917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:57.548986 3932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.549141 3917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:57.549242 3917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "f03e6877589647cbbf586c574a7d3375"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "98a5fa16d3d31dc70597b8c6db13ce5d"
server_key_iv: "5ea0d6a253cb5ff093572d0cbb3f5962"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.549312 3917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:57.577368 3917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:57.579231 3917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:57.579367 3917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:57.583046 3917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33369
I20260928 23:11:57.583084 3984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33369 every 8 connection(s)
I20260928 23:11:57.583406 3917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:57.584233 3985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:57.586445 3985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375: Bootstrap starting.
I20260928 23:11:57.586992 3985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:57.587255 3985 log.cc:824] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:57.587867 3985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375: No bootstrap required, opened a new log
I20260928 23:11:57.589407 3985 raft_consensus.cc:348] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } }
I20260928 23:11:57.589540 3985 raft_consensus.cc:374] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:57.589561 3985 raft_consensus.cc:729] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03e6877589647cbbf586c574a7d3375, State: Initialized, Role: FOLLOWER
I20260928 23:11:57.589717 3985 consensus_queue.cc:260] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } }
I20260928 23:11:57.589778 3985 raft_consensus.cc:388] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:57.589798 3985 raft_consensus.cc:482] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:57.589844 3985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:57.590313 3985 raft_consensus.cc:504] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } }
I20260928 23:11:57.590395 3985 leader_election.cc:302] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f03e6877589647cbbf586c574a7d3375; no voters:
I20260928 23:11:57.590540 3985 leader_election.cc:288] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:57.590549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 3917
I20260928 23:11:57.590579 3988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:57.590634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:57.590804 392 external_mini_cluster.cc:1428] Setting key b28fd03cf9f937ed2fbd92ecf139e477
I20260928 23:11:57.590806 3985 sys_catalog.cc:564] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:57.590905 3988 raft_consensus.cc:686] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [term 1 LEADER]: Becoming Leader. State: Replica: f03e6877589647cbbf586c574a7d3375, State: Running, Role: LEADER
I20260928 23:11:57.590984 3988 consensus_queue.cc:237] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } }
I20260928 23:11:57.591317 3989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f03e6877589647cbbf586c574a7d3375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } } }
I20260928 23:11:57.591393 3989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:57.591553 3990 sys_catalog.cc:455] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f03e6877589647cbbf586c574a7d3375. Latest consensus state: current_term: 1 leader_uuid: "f03e6877589647cbbf586c574a7d3375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f03e6877589647cbbf586c574a7d3375" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33369 } } }
I20260928 23:11:57.591610 3990 sys_catalog.cc:458] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375 [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:57.591871 3999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:11:57.592401 3999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:57.593917 3999 catalog_manager.cc:1347] Generated new cluster ID: ffd68e8d0f334ebe9967160d432fc55a
I20260928 23:11:57.593969 3999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:57.600692 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:57.591316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.62:33369)
0928 23:11:57.591809 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:57.591820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:57.591852 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:11:57.591967 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:57.591975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:57.592184 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:57.592327 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:57.592924 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.594168 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.594852 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.595269 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.595369 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:57.597474 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:57.597477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:57.597477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:57.597486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:57.599375 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.599674 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.599677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.599677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.599758 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.599926 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.599927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.599927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.600155 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:57.600288 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:57.600382 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 23:11:57.600517 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":78,"threads_started":1}
I20260928 23:11:57.613150 3999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:57.613706 3999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:57.618999 3999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f03e6877589647cbbf586c574a7d3375: Generated new TSK 0
I20260928 23:11:57.619150 3999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:57.646095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:57.749938 4011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:57.750154 4011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:57.750176 4011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:57.751348 4011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:57.751394 4011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:57.751421 4011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:57.752929 4011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:57.753175 4011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:57.753403 4011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:11:57.758105 4011 server_base.cc:1034] running on GCE node
W20260928 23:11:57.758095 4016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.758088 4019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.758080 4017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:57.758602 4011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:57.758846 4011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:57.759991 4011 hybrid_clock.cc:648] HybridClock initialized: now 1790637117759969 us; error 34 us; skew 500 ppm
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:57.762454 4011 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:57.762858 4011 webserver.cc:460] Webserver started at http://127.0.98.1:43555/ using document root <none> and password file <none>
I20260928 23:11:57.763032 4011 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:57.763072 4011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:57.763157 4011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:57.764117 4011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "b12e79d077784cf8bdfeb167b73f36ad"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "4a99977b6a23528a50b7b088392811c3"
server_key_iv: "02a86daa63ec34f742dd669da64e8d2a"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.764405 4011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b12e79d077784cf8bdfeb167b73f36ad"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "4a99977b6a23528a50b7b088392811c3"
server_key_iv: "02a86daa63ec34f742dd669da64e8d2a"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.765612 4011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:57.766232 4026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.766377 4011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:57.766453 4011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "b12e79d077784cf8bdfeb167b73f36ad"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "4a99977b6a23528a50b7b088392811c3"
server_key_iv: "02a86daa63ec34f742dd669da64e8d2a"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.766505 4011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:57.783327 4011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:57.785058 4011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:57.785161 4011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:57.785388 4011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:57.785647 4011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:57.785677 4011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.785698 4011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:57.785712 4011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.790818 4011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45491
I20260928 23:11:57.790887 4139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45491 every 8 connection(s)
I20260928 23:11:57.791146 4011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:57.799502 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:57.791815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46797 (local address 127.0.98.62:33369)
0928 23:11:57.791860 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:57.791862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:57.792059 (+ 197us) server_negotiation.cc:407] Connection header received
0928 23:11:57.792984 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:57.792986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:57.792998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:57.793048 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:57.793695 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.794193 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.795128 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.795339 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.795393 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:57.797313 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:57.797315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:57.797315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:57.797325 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:57.798335 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.798678 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.798679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.798680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.798731 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.798930 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.798931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.798931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.799033 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:57.799088 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:57.799339 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 23:11:57.799417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:11:57.799654 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:57.791802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33369 (local address 127.0.98.1:46797)
0928 23:11:57.791947 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:57.791981 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:57.792885 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:57.793119 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:57.793130 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:57.793423 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:57.793611 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:57.793616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.794267 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.794268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:57.795052 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:57.795053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.795404 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.795405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:57.795531 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:57.795923 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:57.795931 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:57.797236 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:57.798403 (+ 1167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:57.798409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:57.798411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:57.798611 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:57.798782 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:57.798783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:57.798783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:57.798877 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:57.799084 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:57.799090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:57.799226 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 23:11:57.799311 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 23:11:57.799454 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260928 23:11:57.800086 4140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33369
I20260928 23:11:57.800176 4140 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:57.800336 4140 heartbeater.cc:511] Master 127.0.98.62:33369 requested a full tablet report, sending...
I20260928 23:11:57.800792 3949 ts_manager.cc:194] Registered new tserver with Master: b12e79d077784cf8bdfeb167b73f36ad (127.0.98.1:45491)
I20260928 23:11:57.801188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4011
I20260928 23:11:57.801303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:57.801474 392 external_mini_cluster.cc:1428] Setting key 60b3bd51400978a07a9d9aa213023be9
I20260928 23:11:57.801640 3949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46797
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:11:57.823655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:57.902758 4145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:57.902961 4145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:57.902982 4145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:57.904086 4145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:57.904124 4145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:57.904148 4145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:57.905567 4145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:57.905755 4145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:57.905941 4145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:57.910645 4153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.910673 4151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:57.910683 4150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:57.910717 4145 server_base.cc:1034] running on GCE node
I20260928 23:11:57.911019 4145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:57.911263 4145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:57.912410 4145 hybrid_clock.cc:648] HybridClock initialized: now 1790637117912396 us; error 28 us; skew 500 ppm
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:57.914781 4145 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:57.915263 4145 webserver.cc:460] Webserver started at http://127.0.98.2:42365/ using document root <none> and password file <none>
I20260928 23:11:57.915462 4145 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:57.915510 4145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:57.915632 4145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:57.916643 4145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8fd67a3eb924ff5bee9deb0d4bdf56f"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "91ed5943621d4b4b489de645091b5a36"
server_key_iv: "e2587c45c8c94c4387465138d043dd0c"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.916972 4145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8fd67a3eb924ff5bee9deb0d4bdf56f"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "91ed5943621d4b4b489de645091b5a36"
server_key_iv: "e2587c45c8c94c4387465138d043dd0c"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.918478 4145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:11:57.919236 4160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.919415 4145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:57.919502 4145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "f8fd67a3eb924ff5bee9deb0d4bdf56f"
format_stamp: "Formatted at 2026-09-28 23:11:57 on dist-test-slave-6vgd"
server_key: "91ed5943621d4b4b489de645091b5a36"
server_key_iv: "e2587c45c8c94c4387465138d043dd0c"
server_key_version: "encryptionkey@0"
I20260928 23:11:57.919595 4145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:57.932963 4145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:57.934897 4145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:57.935034 4145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:57.935252 4145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:57.935540 4145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:57.935573 4145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.935602 4145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:57.935623 4145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:57.941041 4145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35689
I20260928 23:11:57.941102 4273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35689 every 8 connection(s)
I20260928 23:11:57.941428 4145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:11:57 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:57.947870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4145
I20260928 23:11:57.947981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:57.948184 392 external_mini_cluster.cc:1428] Setting key bbc773694837616162b7cc6f2331701c
I20260928 23:11:57.950110 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:57.942060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58081 (local address 127.0.98.62:33369)
0928 23:11:57.942105 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:57.942107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:57.942301 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:11:57.943237 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:57.943238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:57.943248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:57.943302 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:57.943869 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.944351 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.945315 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.945502 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.945579 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:57.947569 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:57.947570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:57.947570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:57.947576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:57.948884 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.949261 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.949262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.949262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.949322 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:57.949551 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:57.949551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:57.949552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:57.949673 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:57.949734 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:57.949943 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 23:11:57.950023 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:11:57.950209 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:57.942042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33369 (local address 127.0.98.2:58081)
0928 23:11:57.942192 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:57.942234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:57.943138 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:57.943364 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:57.943367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:57.943610 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:57.943800 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:57.943805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.944411 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.944412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:57.945245 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:57.945247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:57.945584 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:57.945585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:57.945679 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:57.946144 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:57.946153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:57.947478 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:57.948949 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:57.948954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:57.948956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:57.949171 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:57.949381 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:57.949381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:57.949382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:57.949472 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:57.949724 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:57.949727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:57.949835 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 23:11:57.949909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:11:57.950037 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260928 23:11:57.950623 4274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33369
I20260928 23:11:57.950711 4274 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:57.950870 4274 heartbeater.cc:511] Master 127.0.98.62:33369 requested a full tablet report, sending...
I20260928 23:11:57.951210 3948 ts_manager.cc:194] Registered new tserver with Master: f8fd67a3eb924ff5bee9deb0d4bdf56f (127.0.98.2:35689)
I20260928 23:11:57.951573 3948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58081
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:57.970252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:58.048730 4279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:58.048943 4279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:58.048974 4279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:58.050184 4279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:58.050230 4279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:58.050266 4279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:11:58.051767 4279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:58.052047 4279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:58.052249 4279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:58.057057 4285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:58.057054 4287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:58.057031 4284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:58.057106 4279 server_base.cc:1034] running on GCE node
I20260928 23:11:58.057443 4279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:58.057700 4279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:58.058851 4279 hybrid_clock.cc:648] HybridClock initialized: now 1790637118058832 us; error 33 us; skew 500 ppm
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:58.061292 4279 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:11:58.061699 4279 webserver.cc:460] Webserver started at http://127.0.98.3:33639/ using document root <none> and password file <none>
I20260928 23:11:58.061869 4279 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:58.061908 4279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:58.062016 4279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:58.062980 4279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "10a4e48079d347b09028cf2a36b0654b"
format_stamp: "Formatted at 2026-09-28 23:11:58 on dist-test-slave-6vgd"
server_key: "002d933e40f79038928fcc1ef351e72f"
server_key_iv: "ba7103673eb7755e7ad932e6cff233c4"
server_key_version: "encryptionkey@0"
I20260928 23:11:58.063267 4279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10a4e48079d347b09028cf2a36b0654b"
format_stamp: "Formatted at 2026-09-28 23:11:58 on dist-test-slave-6vgd"
server_key: "002d933e40f79038928fcc1ef351e72f"
server_key_iv: "ba7103673eb7755e7ad932e6cff233c4"
server_key_version: "encryptionkey@0"
I20260928 23:11:58.064468 4279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:58.065142 4294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:58.065310 4279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:58.065395 4279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "10a4e48079d347b09028cf2a36b0654b"
format_stamp: "Formatted at 2026-09-28 23:11:58 on dist-test-slave-6vgd"
server_key: "002d933e40f79038928fcc1ef351e72f"
server_key_iv: "ba7103673eb7755e7ad932e6cff233c4"
server_key_version: "encryptionkey@0"
I20260928 23:11:58.065465 4279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:58.073777 4279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:58.075564 4279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:58.075683 4279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:58.075901 4279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:58.076201 4279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:58.076236 4279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:58.076257 4279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:58.076268 4279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:58.081595 4279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37051
I20260928 23:11:58.081662 4407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37051 every 8 connection(s)
I20260928 23:11:58.081936 4279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:11:58.084558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4279
I20260928 23:11:58.084654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:11:58.084856 392 external_mini_cluster.cc:1428] Setting key 2a07b9146addba12b8a5e634d97bcd05
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:58.091050 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:58.082683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39553 (local address 127.0.98.62:33369)
0928 23:11:58.082742 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:58.082743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:58.082938 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:11:58.083850 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:58.083851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:58.083860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:58.083914 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:58.084545 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.085133 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.086140 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.086341 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.086397 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:58.088464 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:58.088465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:58.088466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:58.088473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:58.089753 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.090137 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.090138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.090139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.090192 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.090404 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.090405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.090405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.090516 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:58.090573 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:58.090842 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 23:11:58.090951 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:11:58.091188 4410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:58.082649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33369 (local address 127.0.98.3:39553)
0928 23:11:58.082816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:58.082857 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:58.083761 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:58.083977 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:58.083981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:58.084246 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:58.084436 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:58.084440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.085234 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.085235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:58.086056 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:58.086058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.086407 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.086408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:58.086537 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:58.087017 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:58.087024 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:58.088372 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:58.089828 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:58.089835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:58.089837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:58.090053 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:58.090246 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:58.090247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:58.090247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:58.090348 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:58.090573 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:58.090579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:58.090719 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 23:11:58.090812 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:11:58.090969 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260928 23:11:58.091620 4408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33369
I20260928 23:11:58.091701 4408 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:58.091866 4408 heartbeater.cc:511] Master 127.0.98.62:33369 requested a full tablet report, sending...
I20260928 23:11:58.092235 3948 ts_manager.cc:194] Registered new tserver with Master: 10a4e48079d347b09028cf2a36b0654b (127.0.98.3:37051)
I20260928 23:11:58.092624 3948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39553
I20260928 23:11:58.096292 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[3904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:11:58.118137 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:58.110765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.0.98.1:45491)
0928 23:11:58.110910 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:58.110914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:58.110922 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:11:58.110947 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:58.110949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:58.110963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:58.111027 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:58.111559 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.112286 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.113008 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.113406 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.113470 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:58.115543 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:58.115552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:58.115552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:58.115559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:58.117239 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.117527 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.117530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.117531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.117597 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.117727 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.117728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.117728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.117848 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:58.117942 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:58.117991 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:11:58.118048 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[3904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:58.125419 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:58.119412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.0.98.62:33369)
0928 23:11:58.119460 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:58.119462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:58.119471 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:11:58.119534 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:58.119535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:58.119543 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:58.119581 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:58.119966 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.120431 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.121061 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:58.121276 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:58.121321 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:58.123312 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:58.123314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:58.123315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:58.123322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:58.124507 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.124782 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.124782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.124783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.124824 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:58.124947 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:58.124948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:58.124948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:58.125076 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:58.125147 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:58.125300 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 23:11:58.125356 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 23:11:58.125625 3948 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514
I20260928 23:11:58.125934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4011
I20260928 23:11:58.129964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4145
I20260928 23:11:58.133741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4279
I20260928 23:11:58.137379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 3917
2026-09-28T23:11:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1746 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[4420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[4420](info): set up 2 sockets
Sep 28 23:11:58 dist-test-slave-6vgd krb5kdc[4420](info): commencing operation
krb5kdc: starting...
W20260928 23:11:59.088837 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:11:59.199443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:59.276433 4433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:59.276612 4433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:59.276641 4433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:59.277745 4433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:11:59.277777 4433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:59.277789 4433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:59.277801 4433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:11:59.277812 4433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:11:59.279261 4433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:59.279438 4433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:59.279598 4433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:59.283926 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:59.283850 4439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:59.284004 4441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:59.284096 4433 server_base.cc:1034] running on GCE node
I20260928 23:11:59.284345 4433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:59.284591 4433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:59.285754 4433 hybrid_clock.cc:648] HybridClock initialized: now 1790637119285744 us; error 32 us; skew 500 ppm
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:59.287930 4433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:11:59.288230 4433 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:59.288277 4433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:59.288367 4433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:59.289345 4433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "4acd239adc334f1b86226ace6263813e"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "8cccab425e2bdff427ff1438fee807d7"
server_key_iv: "fd266498eadc56656bc25504d4b1d506"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.289616 4433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "4acd239adc334f1b86226ace6263813e"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "8cccab425e2bdff427ff1438fee807d7"
server_key_iv: "fd266498eadc56656bc25504d4b1d506"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.290678 4433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:11:59.291290 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.291450 4433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:59.291527 4433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "4acd239adc334f1b86226ace6263813e"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "8cccab425e2bdff427ff1438fee807d7"
server_key_iv: "fd266498eadc56656bc25504d4b1d506"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.291576 4433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:59.304507 4433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:59.306092 4433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:59.306191 4433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:59.309540 4433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38451
I20260928 23:11:59.309566 4499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38451 every 8 connection(s)
I20260928 23:11:59.309798 4433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:11:59.310621 4500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:11:59.312729 4500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: Bootstrap starting.
I20260928 23:11:59.313319 4500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: Neither blocks nor log segments found. Creating new log.
I20260928 23:11:59.313330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4433
I20260928 23:11:59.313414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:11:59.313592 392 external_mini_cluster.cc:1428] Setting key a6e681687401f5de0dd53e12d4c22dfd
I20260928 23:11:59.313627 4500 log.cc:824] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:11:59.314311 4500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: No bootstrap required, opened a new log
I20260928 23:11:59.316154 4500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } }
I20260928 23:11:59.316289 4500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:11:59.316320 4500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acd239adc334f1b86226ace6263813e, State: Initialized, Role: FOLLOWER
I20260928 23:11:59.316437 4500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } }
I20260928 23:11:59.316506 4500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:11:59.316526 4500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:11:59.316568 4500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:11:59.317046 4500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } }
I20260928 23:11:59.317137 4500 leader_election.cc:302] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4acd239adc334f1b86226ace6263813e; no voters:
I20260928 23:11:59.317313 4500 leader_election.cc:288] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:11:59.317350 4505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:11:59.317453 4505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [term 1 LEADER]: Becoming Leader. State: Replica: 4acd239adc334f1b86226ace6263813e, State: Running, Role: LEADER
I20260928 23:11:59.317559 4500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:11:59.317590 4505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } }
I20260928 23:11:59.317951 4506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4acd239adc334f1b86226ace6263813e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } } }
I20260928 23:11:59.318061 4506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:59.318197 4505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acd239adc334f1b86226ace6263813e. Latest consensus state: current_term: 1 leader_uuid: "4acd239adc334f1b86226ace6263813e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd239adc334f1b86226ace6263813e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38451 } } }
I20260928 23:11:59.318265 4505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e [sys.catalog]: This master's current role is: LEADER
I20260928 23:11:59.319209 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:59.319659 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:11:59.322003 4514 catalog_manager.cc:1347] Generated new cluster ID: b4e2bbb260ad42cc83ef4fa3434ecd52
I20260928 23:11:59.322053 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:11:59.323366 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:59.314044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.98.62:38451)
0928 23:11:59.314188 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:59.314197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:59.314230 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:11:59.314667 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:59.314673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:59.314721 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:59.314837 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:59.315418 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.316247 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.317046 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.317618 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.317967 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:59.320042 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:59.320046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:59.320047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:59.320057 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:59.321842 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.322194 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.322197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.322197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.322304 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.322495 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.322496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.322497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.322801 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:59.322957 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:59.323022 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 23:11:59.323163 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260928 23:11:59.336784 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:11:59.337332 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:11:59.340195 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: Generated new TSK 0
I20260928 23:11:59.340358 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:11:59.372027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:59.450462 4526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:59.450676 4526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:59.450707 4526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:59.451822 4526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:59.451860 4526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:59.451895 4526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:11:59.453429 4526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:59.453657 4526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:59.453863 4526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:59.458456 4534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:59.458576 4531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:59.458464 4526 server_base.cc:1034] running on GCE node
W20260928 23:11:59.458482 4532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:59.458917 4526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:59.459134 4526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:59.460297 4526 hybrid_clock.cc:648] HybridClock initialized: now 1790637119460281 us; error 35 us; skew 500 ppm
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:59.462617 4526 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:11:59.462944 4526 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:59.462996 4526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:59.463110 4526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:59.464095 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "8642c834aa26488895cf6200119c5d05"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "1a8a3f73d68ca5c00f928e54fa910f8d"
server_key_iv: "a594155d581d636d3167c2fc229bd94f"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.464423 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8642c834aa26488895cf6200119c5d05"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "1a8a3f73d68ca5c00f928e54fa910f8d"
server_key_iv: "a594155d581d636d3167c2fc229bd94f"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.465600 4526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:11:59.466266 4540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.466408 4526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:11:59.466485 4526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "8642c834aa26488895cf6200119c5d05"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "1a8a3f73d68ca5c00f928e54fa910f8d"
server_key_iv: "a594155d581d636d3167c2fc229bd94f"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.466542 4526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:59.483470 4526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:59.485673 4526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:59.485814 4526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:59.486038 4526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:59.486325 4526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:59.486357 4526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.486385 4526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:59.486409 4526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.491889 4526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42757
I20260928 23:11:59.491940 4653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42757 every 8 connection(s)
I20260928 23:11:59.492192 4526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:11:59.496698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4526
I20260928 23:11:59.496809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:11:59.497059 392 external_mini_cluster.cc:1428] Setting key 30a01559fca68fea25b8a47ed0bb25a7
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:59.501737 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:59.492850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34509 (local address 127.0.98.62:38451)
0928 23:11:59.492929 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:59.492931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:59.493068 (+ 137us) server_negotiation.cc:407] Connection header received
0928 23:11:59.493747 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:59.493748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:59.493758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:59.493811 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:59.494737 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.495351 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.496371 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.496563 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.496612 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:59.498776 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:59.498777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:59.498778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:59.498785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:59.500196 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.500684 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.500685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.500686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.500749 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.501003 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.501004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.501004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.501123 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:59.501181 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:59.501510 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 23:11:59.501602 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:11:59.501754 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:59.492816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38451 (local address 127.0.98.1:34509)
0928 23:11:59.492956 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:59.492991 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:59.493658 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:59.493891 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:59.493894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:59.494144 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:59.494667 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:59.494672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.495417 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.495417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:59.496307 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:59.496310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.496641 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.496642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:59.496786 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:59.497283 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:59.497293 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:59.498668 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:59.500280 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:59.500292 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:59.500294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:59.500612 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:59.500813 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:59.500814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:59.500814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:59.500943 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:59.501187 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:59.501192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:59.501374 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 23:11:59.501467 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 23:11:59.501591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 23:11:59.502161 4654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38451
I20260928 23:11:59.502247 4654 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:59.502384 4654 heartbeater.cc:511] Master 127.0.98.62:38451 requested a full tablet report, sending...
I20260928 23:11:59.502862 4464 ts_manager.cc:194] Registered new tserver with Master: 8642c834aa26488895cf6200119c5d05 (127.0.98.1:42757)
I20260928 23:11:59.503445 4464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34509
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260928 23:11:59.824034 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.326s user 0.002s sys 0.000s
I20260928 23:11:59.824383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:11:59.904175 4659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:11:59.904397 4659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:11:59.904421 4659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:11:59.905583 4659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:11:59.905620 4659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:11:59.905645 4659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:11:59.907126 4659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:11:59.907330 4659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:11:59.907521 4659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:11:59.912269 4667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:11:59.912290 4664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:59.912292 4659 server_base.cc:1034] running on GCE node
W20260928 23:11:59.912391 4665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:11:59.912665 4659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:11:59.912887 4659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:11:59.914024 4659 hybrid_clock.cc:648] HybridClock initialized: now 1790637119914009 us; error 30 us; skew 500 ppm
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:11:59.922766 4659 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:11:59.923112 4659 fs_manager.cc:359] Metadata directory not provided
I20260928 23:11:59.923163 4659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:11:59.923286 4659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:11:59.924347 4659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "54338c5dad8a417da83934729114483b"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "aa33eb74f50ac0e574d9928d12cf99b0"
server_key_iv: "3a6b0a929824c0433690ef071215608c"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.924677 4659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54338c5dad8a417da83934729114483b"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "aa33eb74f50ac0e574d9928d12cf99b0"
server_key_iv: "3a6b0a929824c0433690ef071215608c"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.925948 4659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:11:59.926625 4673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.926771 4659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:11:59.926851 4659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "54338c5dad8a417da83934729114483b"
format_stamp: "Formatted at 2026-09-28 23:11:59 on dist-test-slave-6vgd"
server_key: "aa33eb74f50ac0e574d9928d12cf99b0"
server_key_iv: "3a6b0a929824c0433690ef071215608c"
server_key_version: "encryptionkey@0"
I20260928 23:11:59.926924 4659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:11:59.954423 4659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:11:59.956207 4659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:11:59.956339 4659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:11:59.956547 4659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:11:59.956836 4659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:11:59.956867 4659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.956897 4659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:11:59.956928 4659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:11:59.962128 4659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43671
I20260928 23:11:59.962172 4786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43671 every 8 connection(s)
I20260928 23:11:59.962409 4659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:11:59 dist-test-slave-6vgd krb5kdc[4420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:11:59.969200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4659
I20260928 23:11:59.969311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:11:59.969501 392 external_mini_cluster.cc:1428] Setting key 8019c15edf20eacf5ef3b8a738e5b39a
I20260928 23:11:59.970976 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:59.963065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59433 (local address 127.0.98.62:38451)
0928 23:11:59.963109 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:11:59.963111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:11:59.963279 (+ 168us) server_negotiation.cc:407] Connection header received
0928 23:11:59.963733 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:11:59.963734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:11:59.963743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:11:59.963798 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:11:59.964681 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.965182 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.966138 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.966335 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.966379 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:59.968263 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:11:59.968265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:11:59.968265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:11:59.968272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:11:59.969665 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.970128 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.970128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.970129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.970193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:11:59.970410 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:11:59.970411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:11:59.970411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:11:59.970531 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:11:59.970597 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:11:59.970809 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:11:59.970879 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:11:59.971095 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:11:59.963032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38451 (local address 127.0.98.2:59433)
0928 23:11:59.963172 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:11:59.963212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:11:59.963658 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:11:59.963853 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:11:59.963856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:11:59.964114 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:11:59.964596 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:59.964601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.965276 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.965277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:59.966071 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:11:59.966073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:11:59.966401 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:11:59.966402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:11:59.966505 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:11:59.966927 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:11:59.966934 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:11:59.968183 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:11:59.969782 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:59.969788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:59.969790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:59.970044 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:59.970250 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:11:59.970251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:11:59.970251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:11:59.970356 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:11:59.970585 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:11:59.970589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:11:59.970704 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 23:11:59.970779 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:11:59.970901 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 23:11:59.971599 4787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38451
I20260928 23:11:59.971695 4787 heartbeater.cc:464] Registering TS with master...
I20260928 23:11:59.971841 4787 heartbeater.cc:511] Master 127.0.98.62:38451 requested a full tablet report, sending...
I20260928 23:11:59.972163 4464 ts_manager.cc:194] Registered new tserver with Master: 54338c5dad8a417da83934729114483b (127.0.98.2:43671)
I20260928 23:11:59.972518 4464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59433
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:11:59.991844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:00.069182 4792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:00.069394 4792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:00.069417 4792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:00.070585 4792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:00.070627 4792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:00.070653 4792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:00.072124 4792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:00.072341 4792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:00.072548 4792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:00.077086 4798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.077126 4797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.077095 4800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:00.077299 4792 server_base.cc:1034] running on GCE node
I20260928 23:12:00.077486 4792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:00.077689 4792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:00.078815 4792 hybrid_clock.cc:648] HybridClock initialized: now 1790637120078806 us; error 26 us; skew 500 ppm
Sep 28 23:12:00 dist-test-slave-6vgd krb5kdc[4420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:00.081056 4792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:12:00.081408 4792 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:00.081457 4792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:00.081568 4792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:00.082568 4792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "5da61ec0dea04d8888514569b34d2422"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "1de57dda812b153d619102f9b20d0d56"
server_key_iv: "b56b6ff96518318acc1ba44a608ce397"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.082875 4792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5da61ec0dea04d8888514569b34d2422"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "1de57dda812b153d619102f9b20d0d56"
server_key_iv: "b56b6ff96518318acc1ba44a608ce397"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.084022 4792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:00.084621 4806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.084753 4792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:00.084829 4792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "5da61ec0dea04d8888514569b34d2422"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "1de57dda812b153d619102f9b20d0d56"
server_key_iv: "b56b6ff96518318acc1ba44a608ce397"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.084887 4792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:00.094761 4792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:00.096457 4792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:00.096585 4792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:00.096773 4792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:00.097059 4792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:00.097091 4792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.097119 4792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:00.097142 4792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.102392 4792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45583
I20260928 23:12:00.102435 4919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45583 every 8 connection(s)
I20260928 23:12:00.102667 4792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:00.105938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4792
I20260928 23:12:00.106036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:00.106206 392 external_mini_cluster.cc:1428] Setting key 37cf57f0ab013f174bbb28d39827277c
Sep 28 23:12:00 dist-test-slave-6vgd krb5kdc[4420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:00.111126 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.103293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45693 (local address 127.0.98.62:38451)
0928 23:12:00.103341 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.103343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.103535 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:12:00.104026 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.104028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.104040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.104086 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:00.104994 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.105523 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.106586 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.106772 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.106826 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.108688 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:00.108690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:00.108691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:00.108698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:00.109965 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:00.110303 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:00.110304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:00.110304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:00.110353 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:00.110561 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:00.110561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:00.110562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:00.110664 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:00.110719 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.110964 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.111038 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:12:00.111284 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.103261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38451 (local address 127.0.98.3:45693)
0928 23:12:00.103427 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.103461 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.103933 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.104157 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.104161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.104420 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:00.104899 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.104904 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.105599 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.105600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.106512 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.106514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.106836 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.106837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.106962 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.107360 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:00.107367 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:00.108608 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:00.110036 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:00.110042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:00.110044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:00.110241 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:00.110403 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:00.110404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:00.110404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:00.110508 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:00.110715 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:00.110721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:00.110850 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.110933 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.111080 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260928 23:12:00.111693 4920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38451
I20260928 23:12:00.111776 4920 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:00.111907 4920 heartbeater.cc:511] Master 127.0.98.62:38451 requested a full tablet report, sending...
I20260928 23:12:00.112217 4464 ts_manager.cc:194] Registered new tserver with Master: 5da61ec0dea04d8888514569b34d2422 (127.0.98.3:45583)
I20260928 23:12:00.112577 4464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45693
I20260928 23:12:00.117657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:12:00.124176 4504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.118901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.0.98.62:38451)
0928 23:12:00.118979 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.118981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.119034 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:12:00.119077 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.119077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.119086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.119128 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:00.119616 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.120143 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.120801 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.120979 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.121020 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.122093 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:00.122094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:00.122095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:00.122101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:00.123359 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:00.123614 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:00.123615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:00.123615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:00.123661 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:00.123769 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:00.123770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:00.123770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:00.123905 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:00.123978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.124046 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.124102 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 23:12:00.125487 4464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:00.127059 4464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:00.133247 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45583 (local address 127.0.0.1:35980)
0928 23:12:00.129948 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.129957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.130002 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.130221 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.130222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.130224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:00.130266 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.130267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131148 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.131149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.132208 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.132210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.133000 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.133001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.133052 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.133063 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.133116 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.133151 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":88,"thread_start_us":56,"threads_started":1}
I20260928 23:12:00.133265 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.0.98.3:45583)
0928 23:12:00.129868 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.129873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.130056 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:12:00.130104 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.130107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.130121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.130191 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:00.130319 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131065 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.132277 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.132920 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.132998 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.133056 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.133128 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.133164 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 23:12:00.133782 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.0.98.1:42757)
0928 23:12:00.129757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.129762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.129914 (+ 152us) server_negotiation.cc:407] Connection header received
0928 23:12:00.129948 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.129950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.129966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.130032 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:00.130220 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131304 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.132511 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.133186 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.133284 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.133340 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.133667 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.133706 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":41,"threads_started":1}
I20260928 23:12:00.134152 4854 tablet_service.cc:1467] Processing CreateTablet for tablet 561670212df040ed92bd4227d00bd0c1 (DEFAULT_TABLE table=test-table [id=82cbb10f828642fe9c1cb05bf9759c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:00.134342 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43671 (local address 127.0.0.1:45910)
0928 23:12:00.129372 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.129412 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.129488 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.129778 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.129781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.129784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:00.129914 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.129916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131194 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.131195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.132427 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.132429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.134121 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.134122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.134188 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.134192 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.134239 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.134278 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":6,"thread_start_us":53,"threads_started":1}
I20260928 23:12:00.134526 4854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561670212df040ed92bd4227d00bd0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:00.134550 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.98.2:43671)
0928 23:12:00.129461 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.129467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.129477 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:12:00.129578 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.129581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.129597 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.129677 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:00.130007 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131108 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.133373 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.134058 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.134121 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.134172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.134439 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.134478 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260928 23:12:00.136953 4937 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Bootstrap starting.
I20260928 23:12:00.137122 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.129488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42757 (local address 127.0.0.1:34046)
0928 23:12:00.129803 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.129812 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.129863 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.130094 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.130095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.130097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:00.130129 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.130130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.131361 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.131361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.132444 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.132445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.133286 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.133287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.133585 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.133591 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.133653 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.133688 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":69,"threads_started":1}
I20260928 23:12:00.137717 4937 tablet_bootstrap.cc:654] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:00.137593 4721 tablet_service.cc:1467] Processing CreateTablet for tablet 561670212df040ed92bd4227d00bd0c1 (DEFAULT_TABLE table=test-table [id=82cbb10f828642fe9c1cb05bf9759c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:00.137861 4588 tablet_service.cc:1467] Processing CreateTablet for tablet 561670212df040ed92bd4227d00bd0c1 (DEFAULT_TABLE table=test-table [id=82cbb10f828642fe9c1cb05bf9759c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:00.137948 4721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561670212df040ed92bd4227d00bd0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:00.138136 4588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561670212df040ed92bd4227d00bd0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:00.138270 4937 log.cc:824] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:00.139012 4937 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: No bootstrap required, opened a new log
I20260928 23:12:00.139084 4937 ts_tablet_manager.cc:1397] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:00.140491 4939 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Bootstrap starting.
I20260928 23:12:00.141013 4937 raft_consensus.cc:348] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.141175 4939 tablet_bootstrap.cc:654] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:00.141171 4937 raft_consensus.cc:374] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:00.141206 4937 raft_consensus.cc:729] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da61ec0dea04d8888514569b34d2422, State: Initialized, Role: FOLLOWER
I20260928 23:12:00.141342 4937 consensus_queue.cc:260] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.141489 4939 log.cc:824] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:00.141607 4920 heartbeater.cc:503] Master 127.0.98.62:38451 was elected leader, sending a full tablet report...
I20260928 23:12:00.141860 4937 ts_tablet_manager.cc:1428] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 23:12:00.142058 4940 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Bootstrap starting.
I20260928 23:12:00.142206 4939 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: No bootstrap required, opened a new log
I20260928 23:12:00.142300 4939 ts_tablet_manager.cc:1397] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:00.142736 4940 tablet_bootstrap.cc:654] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:00.143216 4940 log.cc:824] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:00.143874 4940 tablet_bootstrap.cc:492] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: No bootstrap required, opened a new log
I20260928 23:12:00.143926 4940 ts_tablet_manager.cc:1397] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:00.144246 4939 raft_consensus.cc:348] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.144366 4939 raft_consensus.cc:374] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:00.144395 4939 raft_consensus.cc:729] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8642c834aa26488895cf6200119c5d05, State: Initialized, Role: FOLLOWER
I20260928 23:12:00.144502 4939 consensus_queue.cc:260] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.144732 4939 ts_tablet_manager.cc:1428] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:00.144784 4654 heartbeater.cc:503] Master 127.0.98.62:38451 was elected leader, sending a full tablet report...
I20260928 23:12:00.145551 4940 raft_consensus.cc:348] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.145711 4940 raft_consensus.cc:374] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:00.145751 4940 raft_consensus.cc:729] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54338c5dad8a417da83934729114483b, State: Initialized, Role: FOLLOWER
I20260928 23:12:00.145834 4940 consensus_queue.cc:260] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.146023 4940 ts_tablet_manager.cc:1428] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:00.146057 4787 heartbeater.cc:503] Master 127.0.98.62:38451 was elected leader, sending a full tablet report...
W20260928 23:12:00.212910 4788 tablet.cc:2369] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:12:00.243527 4655 tablet.cc:2369] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:12:00.353188 4921 tablet.cc:2369] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:12:00.486584 4944 raft_consensus.cc:482] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:00.486723 4944 raft_consensus.cc:504] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.487076 4944 leader_election.cc:288] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54338c5dad8a417da83934729114483b (127.0.98.2:43671), 5da61ec0dea04d8888514569b34d2422 (127.0.98.3:45583)
I20260928 23:12:00.491091 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.487297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56627 (local address 127.0.98.2:43671)
0928 23:12:00.487349 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.487351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.487494 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:12:00.487556 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.487556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.487568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.487646 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:00.487815 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.488568 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490157 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.490738 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490818 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.490829 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.490987 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.491020 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:12:00.491091 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.487561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43307 (local address 127.0.98.3:45583)
0928 23:12:00.487607 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.487609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.487638 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:12:00.487691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.487692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.487701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.487742 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:00.487904 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.488573 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490162 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.490738 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490818 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.490829 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.490972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.491020 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:12:00.491115 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.487243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43671 (local address 127.0.98.1:56627)
0928 23:12:00.487409 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.487419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.487496 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.487652 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.487653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.487654 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:00.487735 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.487738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.488661 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.488661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.490045 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.490048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.490820 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.490905 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.490910 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.490979 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.491006 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":73,"thread_start_us":106,"threads_started":1}
I20260928 23:12:00.491117 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.487467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45583 (local address 127.0.98.1:43307)
0928 23:12:00.487596 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.487604 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.487647 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.487758 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.487759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.487760 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:00.487809 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.487811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.488644 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.488645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.490045 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:00.490048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.490819 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.490820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:00.490905 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.490910 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.490962 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.491003 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260928 23:12:00.491465 4741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561670212df040ed92bd4227d00bd0c1" candidate_uuid: "8642c834aa26488895cf6200119c5d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54338c5dad8a417da83934729114483b" is_pre_election: true
I20260928 23:12:00.491616 4741 raft_consensus.cc:2454] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8642c834aa26488895cf6200119c5d05 in term 0.
I20260928 23:12:00.491641 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561670212df040ed92bd4227d00bd0c1" candidate_uuid: "8642c834aa26488895cf6200119c5d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da61ec0dea04d8888514569b34d2422" is_pre_election: true
I20260928 23:12:00.491796 4874 raft_consensus.cc:2454] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8642c834aa26488895cf6200119c5d05 in term 0.
I20260928 23:12:00.491832 4544 leader_election.cc:302] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54338c5dad8a417da83934729114483b, 8642c834aa26488895cf6200119c5d05; no voters:
I20260928 23:12:00.491947 4944 raft_consensus.cc:2781] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:00.492008 4944 raft_consensus.cc:482] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:00.492031 4944 raft_consensus.cc:3037] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:00.492678 4944 raft_consensus.cc:504] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.492771 4944 leader_election.cc:288] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [CANDIDATE]: Term 1 election: Requested vote from peers 54338c5dad8a417da83934729114483b (127.0.98.2:43671), 5da61ec0dea04d8888514569b34d2422 (127.0.98.3:45583)
I20260928 23:12:00.492954 4874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561670212df040ed92bd4227d00bd0c1" candidate_uuid: "8642c834aa26488895cf6200119c5d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da61ec0dea04d8888514569b34d2422"
I20260928 23:12:00.492969 4741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561670212df040ed92bd4227d00bd0c1" candidate_uuid: "8642c834aa26488895cf6200119c5d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54338c5dad8a417da83934729114483b"
I20260928 23:12:00.493011 4874 raft_consensus.cc:3037] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:00.493042 4741 raft_consensus.cc:3037] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:00.493760 4874 raft_consensus.cc:2454] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8642c834aa26488895cf6200119c5d05 in term 1.
I20260928 23:12:00.493883 4741 raft_consensus.cc:2454] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8642c834aa26488895cf6200119c5d05 in term 1.
I20260928 23:12:00.493902 4544 leader_election.cc:302] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da61ec0dea04d8888514569b34d2422, 8642c834aa26488895cf6200119c5d05; no voters:
I20260928 23:12:00.493974 4944 raft_consensus.cc:2781] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:00.494138 4944 raft_consensus.cc:686] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [term 1 LEADER]: Becoming Leader. State: Replica: 8642c834aa26488895cf6200119c5d05, State: Running, Role: LEADER
I20260928 23:12:00.494246 4944 consensus_queue.cc:237] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } }
I20260928 23:12:00.495097 4464 catalog_manager.cc:5495] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8642c834aa26488895cf6200119c5d05 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8642c834aa26488895cf6200119c5d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8642c834aa26488895cf6200119c5d05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 } health_report { overall_health: UNKNOWN } } }
I20260928 23:12:00.570438 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.568096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.98.1:42757)
0928 23:12:00.568145 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.568147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.568156 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:00.568219 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.568220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.568240 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.568301 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:00.568408 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.568866 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.569725 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:00.569913 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:00.569967 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:00.570102 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:00.570240 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:00.570311 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.570330 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.570359 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:12:00.572122 4741 raft_consensus.cc:1261] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 1 FOLLOWER]: Refusing update from remote peer 8642c834aa26488895cf6200119c5d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:00.572120 4874 raft_consensus.cc:1261] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 1 FOLLOWER]: Refusing update from remote peer 8642c834aa26488895cf6200119c5d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:00.572451 4944 consensus_queue.cc:1035] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54338c5dad8a417da83934729114483b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:00.572568 4948 consensus_queue.cc:1035] T 561670212df040ed92bd4227d00bd0c1 P 8642c834aa26488895cf6200119c5d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da61ec0dea04d8888514569b34d2422" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:00.575063 4953 mvcc.cc:205] Tried to move back new op lower bound from 7334449645860925440 to 7334449645544906752. Current Snapshot: MvccSnapshot[applied={T|T < 7334449645860925440}]
I20260928 23:12:00.576776 4463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:00.576864 4463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:00.577571 4463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4acd239adc334f1b86226ace6263813e: Sending DeleteTablet for 3 replicas of tablet 561670212df040ed92bd4227d00bd0c1
I20260928 23:12:00.577865 4721 tablet_service.cc:1514] Processing DeleteTablet for tablet 561670212df040ed92bd4227d00bd0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45910
I20260928 23:12:00.577953 4588 tablet_service.cc:1514] Processing DeleteTablet for tablet 561670212df040ed92bd4227d00bd0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34046
I20260928 23:12:00.578099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4526
I20260928 23:12:00.578158 4854 tablet_service.cc:1514] Processing DeleteTablet for tablet 561670212df040ed92bd4227d00bd0c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35980
I20260928 23:12:00.579540 4963 tablet_replica.cc:330] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: stopping tablet replica
I20260928 23:12:00.579628 4963 raft_consensus.cc:2229] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:00.579712 4963 raft_consensus.cc:2258] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:00.579854 4961 tablet_replica.cc:330] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: stopping tablet replica
I20260928 23:12:00.579931 4961 raft_consensus.cc:2229] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:00.580025 4961 raft_consensus.cc:2258] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:00.580120 4963 ts_tablet_manager.cc:1905] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:00.580540 4961 ts_tablet_manager.cc:1905] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:00.581882 4963 ts_tablet_manager.cc:1918] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:00.581946 4963 log.cc:1196] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/561670212df040ed92bd4227d00bd0c1
I20260928 23:12:00.582006 4961 ts_tablet_manager.cc:1918] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:00.582074 4961 log.cc:1196] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/561670212df040ed92bd4227d00bd0c1
I20260928 23:12:00.582218 4963 ts_tablet_manager.cc:1939] T 561670212df040ed92bd4227d00bd0c1 P 5da61ec0dea04d8888514569b34d2422: Deleting consensus metadata
I20260928 23:12:00.582340 4961 ts_tablet_manager.cc:1939] T 561670212df040ed92bd4227d00bd0c1 P 54338c5dad8a417da83934729114483b: Deleting consensus metadata
I20260928 23:12:00.582628 4451 catalog_manager.cc:4841] TS 5da61ec0dea04d8888514569b34d2422 (127.0.98.3:45583): tablet 561670212df040ed92bd4227d00bd0c1 (table test-table [id=82cbb10f828642fe9c1cb05bf9759c74]) successfully deleted
I20260928 23:12:00.582723 4451 catalog_manager.cc:4841] TS 54338c5dad8a417da83934729114483b (127.0.98.2:43671): tablet 561670212df040ed92bd4227d00bd0c1 (table test-table [id=82cbb10f828642fe9c1cb05bf9759c74]) successfully deleted
W20260928 23:12:00.583992 4451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42757 (error 108)
I20260928 23:12:00.584105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4659
I20260928 23:12:00.584857 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.584736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42757 (local address 127.0.0.1:34062)
0928 23:12:00.584789 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.584823 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 23:12:00.584961 4451 catalog_manager.cc:4584] TS 8642c834aa26488895cf6200119c5d05 (127.0.98.1:42757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 561670212df040ed92bd4227d00bd0c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42757: connect: Connection refused (error 111)
I20260928 23:12:00.588718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4792
I20260928 23:12:00.592197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4433
2026-09-28T23:12:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2454 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T23:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:00Z Disabled control of system clock
I20260928 23:12:00.616557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:12:00.693610 4968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:00.693805 4968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:00.693826 4968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:00.694913 4968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:00.694945 4968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:00.694957 4968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:00.694968 4968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:00.694993 4968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:00.696245 4968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:00.696425 4968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:00.696657 4968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:00.701058 4974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.701068 4976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:00.701156 4968 server_base.cc:1034] running on GCE node
W20260928 23:12:00.701050 4973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:00.701478 4968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:00.701725 4968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:00.702853 4968 hybrid_clock.cc:648] HybridClock initialized: now 1790637120702840 us; error 29 us; skew 500 ppm
I20260928 23:12:00.703943 4968 webserver.cc:460] Webserver started at http://127.0.98.62:39871/ using document root <none> and password file <none>
I20260928 23:12:00.704171 4968 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:00.704216 4968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:00.704337 4968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:00.705374 4968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "df89611cd6b644a0b5f0dbb3f7c0086e"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "63336fc52d614b99b9b28ebaa4b77b86"
server_key_iv: "6233ec1b0cb386f7941e9ae8e4a73c58"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.705686 4968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "df89611cd6b644a0b5f0dbb3f7c0086e"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "63336fc52d614b99b9b28ebaa4b77b86"
server_key_iv: "6233ec1b0cb386f7941e9ae8e4a73c58"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.706811 4968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:00.707399 4982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.707538 4968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:00.707614 4968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "df89611cd6b644a0b5f0dbb3f7c0086e"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "63336fc52d614b99b9b28ebaa4b77b86"
server_key_iv: "6233ec1b0cb386f7941e9ae8e4a73c58"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.707671 4968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:00.710889 4968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:00.711122 4968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:00.711234 4968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:00.714706 4968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34489
I20260928 23:12:00.714735 5034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34489 every 8 connection(s)
I20260928 23:12:00.715013 4968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:00.715588 5035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:00.718187 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: Bootstrap starting.
I20260928 23:12:00.718712 5035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:00.718948 5035 log.cc:824] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:00.719499 5035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: No bootstrap required, opened a new log
I20260928 23:12:00.720007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 4968
I20260928 23:12:00.720091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:00.720244 392 external_mini_cluster.cc:1428] Setting key 491945ef074b61b39398a4908e9d51ac
I20260928 23:12:00.721441 5035 raft_consensus.cc:348] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260928 23:12:00.721604 5035 raft_consensus.cc:374] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:00.721675 5035 raft_consensus.cc:729] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df89611cd6b644a0b5f0dbb3f7c0086e, State: Initialized, Role: FOLLOWER
I20260928 23:12:00.721820 5035 consensus_queue.cc:260] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260928 23:12:00.721930 5035 raft_consensus.cc:388] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:00.721963 5035 raft_consensus.cc:482] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:00.722028 5035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:00.722596 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.720708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.0.98.62:34489)
0928 23:12:00.720919 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.720928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.720962 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:12:00.721407 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.721416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.721475 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.721621 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:00.722175 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:00.722179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:00.722179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:00.722183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:00.722224 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:00.722238 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:00.722306 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.722340 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.722403 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260928 23:12:00.722666 5035 raft_consensus.cc:504] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260928 23:12:00.722759 5035 leader_election.cc:302] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df89611cd6b644a0b5f0dbb3f7c0086e; no voters:
I20260928 23:12:00.722918 5035 leader_election.cc:288] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:00.723133 5035 sys_catalog.cc:564] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:00.723163 5040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:00.723322 5040 raft_consensus.cc:686] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [term 1 LEADER]: Becoming Leader. State: Replica: df89611cd6b644a0b5f0dbb3f7c0086e, State: Running, Role: LEADER
I20260928 23:12:00.723404 5040 consensus_queue.cc:237] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260928 23:12:00.723717 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [sys.catalog]: SysCatalogTable state changed. Reason: New leader df89611cd6b644a0b5f0dbb3f7c0086e. Latest consensus state: current_term: 1 leader_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260928 23:12:00.723798 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:00.723726 5041 sys_catalog.cc:455] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df89611cd6b644a0b5f0dbb3f7c0086e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260928 23:12:00.723886 5041 sys_catalog.cc:458] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:00.724052 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:00.724556 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:00.725943 5049 catalog_manager.cc:1347] Generated new cluster ID: 6f68579db04f4464bab24295dba616e0
I20260928 23:12:00.725986 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:00.735824 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:00.735996 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:00.741967 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: Generated new TSK 0
I20260928 23:12:00.742175 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:12:00.752689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 23:12:00.834419 5059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:00.834620 5059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:00.834641 5059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:00.835812 5059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:00.835855 5059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:00.835879 5059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:00.837184 5059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:00.837428 5059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:00.837713 5059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:00.842367 5067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.842402 5064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.842641 5065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:00.842679 5059 server_base.cc:1034] running on GCE node
I20260928 23:12:00.842890 5059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:00.843109 5059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:00.844255 5059 hybrid_clock.cc:648] HybridClock initialized: now 1790637120844241 us; error 30 us; skew 500 ppm
I20260928 23:12:00.845434 5059 webserver.cc:460] Webserver started at http://127.0.98.1:43393/ using document root <none> and password file <none>
I20260928 23:12:00.845664 5059 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:00.845708 5059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:00.845831 5059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:00.846851 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "45e7249eb69f47ff8ad0cc3853e2b211"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "8f795e349a0636d77a3cf882a3382787"
server_key_iv: "3ba671208b5721b4720d01fecf5f867a"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.847167 5059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "45e7249eb69f47ff8ad0cc3853e2b211"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "8f795e349a0636d77a3cf882a3382787"
server_key_iv: "3ba671208b5721b4720d01fecf5f867a"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.848321 5059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:00.848928 5073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.849056 5059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:00.849133 5059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "45e7249eb69f47ff8ad0cc3853e2b211"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "8f795e349a0636d77a3cf882a3382787"
server_key_iv: "3ba671208b5721b4720d01fecf5f867a"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.849195 5059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:00.852322 5059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:00.852578 5059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:00.852696 5059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:00.852890 5059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:00.853168 5059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:00.853199 5059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.853245 5059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:00.853271 5059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.858564 5059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43557
I20260928 23:12:00.858600 5186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43557 every 8 connection(s)
I20260928 23:12:00.858891 5059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:00.859730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5059
I20260928 23:12:00.859833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:00.859997 392 external_mini_cluster.cc:1428] Setting key a553741eb02c1cfd5016d2a889120dad
I20260928 23:12:00.860852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:12:00.861930 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.859543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33095 (local address 127.0.98.62:34489)
0928 23:12:00.859608 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.859610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.859830 (+ 220us) server_negotiation.cc:407] Connection header received
0928 23:12:00.860341 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.860342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.860348 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.860408 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:00.861650 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:00.861651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:00.861651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:00.861655 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:00.861673 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:00.861678 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:00.861740 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.861801 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.861817 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 23:12:00.862018 5189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.859540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.1:33095)
0928 23:12:00.859704 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.859743 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.860246 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.860479 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.860483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.860492 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:00.861500 (+ 1008us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:00.861500 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:00.861531 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:00.861560 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:00.861740 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:00.861743 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:00.861745 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.861795 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.861840 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260928 23:12:00.862512 5187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260928 23:12:00.862596 5187 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:00.862733 5187 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260928 23:12:00.863224 4999 ts_manager.cc:194] Registered new tserver with Master: 45e7249eb69f47ff8ad0cc3853e2b211 (127.0.98.1:43557)
W20260928 23:12:00.938655 5190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:00.938850 5190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:00.938870 5190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:00.939939 5190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:00.939971 5190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:00.939996 5190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:00.941293 5190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:00.941473 5190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:00.941706 5190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:00.946237 5198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.946260 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:00.946417 5196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:00.946420 5190 server_base.cc:1034] running on GCE node
I20260928 23:12:00.946681 5190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:00.946887 5190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:00.948029 5190 hybrid_clock.cc:648] HybridClock initialized: now 1790637120948017 us; error 29 us; skew 500 ppm
I20260928 23:12:00.949101 5190 webserver.cc:460] Webserver started at http://127.0.98.2:36745/ using document root <none> and password file <none>
I20260928 23:12:00.949337 5190 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:00.949385 5190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:00.949527 5190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:00.950515 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "a210c45165f04ac8ac96770542791910"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "03d0d53e487a82062073a6c4cd555af6"
server_key_iv: "211e0b54b8eb0ef6f01986dfcfc437ef"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.950856 5190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a210c45165f04ac8ac96770542791910"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "03d0d53e487a82062073a6c4cd555af6"
server_key_iv: "211e0b54b8eb0ef6f01986dfcfc437ef"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.952064 5190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:00.952713 5204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.952868 5190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:00.952944 5190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "a210c45165f04ac8ac96770542791910"
format_stamp: "Formatted at 2026-09-28 23:12:00 on dist-test-slave-6vgd"
server_key: "03d0d53e487a82062073a6c4cd555af6"
server_key_iv: "211e0b54b8eb0ef6f01986dfcfc437ef"
server_key_version: "encryptionkey@0"
I20260928 23:12:00.953007 5190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:00.956161 5190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:00.956388 5190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:00.956506 5190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:00.956689 5190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:00.956964 5190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:00.956996 5190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.957026 5190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:00.957048 5190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:00.962327 5190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36847
I20260928 23:12:00.962387 5317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36847 every 8 connection(s)
I20260928 23:12:00.962667 5190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:00.965256 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.963303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43309 (local address 127.0.98.62:34489)
0928 23:12:00.963363 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:00.963365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:00.963566 (+ 201us) server_negotiation.cc:407] Connection header received
0928 23:12:00.964001 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:00.964002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:00.964006 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:00.964062 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:00.965051 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:00.965052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:00.965052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:00.965056 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:00.965070 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:00.965073 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:00.965115 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:00.965164 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:12:00.965175 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:12:00.965415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5190
I20260928 23:12:00.965459 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:00.963284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.2:43309)
0928 23:12:00.963451 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:00.963490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:00.963928 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:00.964117 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:00.964120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:00.964127 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:00.964921 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:00.964921 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:00.964949 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:00.964976 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:00.965113 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:00.965116 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:00.965118 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:00.965165 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:12:00.965234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260928 23:12:00.965512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:00.965678 392 external_mini_cluster.cc:1428] Setting key 29faff146250a82c0a598ceee77f70dc
I20260928 23:12:00.966126 5318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260928 23:12:00.966220 5318 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:00.966363 5318 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260928 23:12:00.966578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 23:12:00.966692 4999 ts_manager.cc:194] Registered new tserver with Master: a210c45165f04ac8ac96770542791910 (127.0.98.2:36847)
W20260928 23:12:01.043855 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:01.044051 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:01.044071 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:01.045195 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:01.045248 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:01.045276 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:01.046589 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:01.046806 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:01.047060 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:01.051558 5329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.051575 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.051574 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.051751 5321 server_base.cc:1034] running on GCE node
I20260928 23:12:01.052012 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:01.052232 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:01.053361 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790637121053346 us; error 29 us; skew 500 ppm
I20260928 23:12:01.054406 5321 webserver.cc:460] Webserver started at http://127.0.98.3:45039/ using document root <none> and password file <none>
I20260928 23:12:01.054615 5321 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:01.054659 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:01.054783 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:01.055824 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "76e0a703c7714061818f62e2a02f12e9"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "76567f0dca59c1b516a99a59c2d4503f"
server_key_iv: "6f1ffbe6b5e5b0eb18eab8f7b3a693e6"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.056181 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76e0a703c7714061818f62e2a02f12e9"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "76567f0dca59c1b516a99a59c2d4503f"
server_key_iv: "6f1ffbe6b5e5b0eb18eab8f7b3a693e6"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.057402 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:12:01.058043 5335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.058187 5321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:01.058260 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "76e0a703c7714061818f62e2a02f12e9"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "76567f0dca59c1b516a99a59c2d4503f"
server_key_iv: "6f1ffbe6b5e5b0eb18eab8f7b3a693e6"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.058323 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:01.061430 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:01.061678 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:01.061790 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:01.061992 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:01.062258 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:01.062288 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.062312 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:01.062337 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.067513 5321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33113
I20260928 23:12:01.067548 5448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33113 every 8 connection(s)
I20260928 23:12:01.067817 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:01.070415 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.068450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40281 (local address 127.0.98.62:34489)
0928 23:12:01.068502 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.068504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.068702 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:12:01.069108 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.069109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.069115 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.069163 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.070227 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.070227 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.070228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.070230 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.070241 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.070244 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.070283 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.070350 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.070361 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:12:01.070456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5321
I20260928 23:12:01.070551 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:01.070638 5451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.068441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.3:40281)
0928 23:12:01.068596 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.068625 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.069035 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.069236 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.069240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.069250 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.070105 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.070106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.070133 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.070159 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.070284 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.070286 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.070289 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.070361 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.070422 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260928 23:12:01.070744 392 external_mini_cluster.cc:1428] Setting key 5c7c5527e073eb9f3c83b073e8fe7a15
I20260928 23:12:01.071017 5449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260928 23:12:01.071106 5449 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:01.071249 5449 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260928 23:12:01.071605 4999 ts_manager.cc:194] Registered new tserver with Master: 76e0a703c7714061818f62e2a02f12e9 (127.0.98.3:33113)
I20260928 23:12:01.082298 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:12:01.084542 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.083515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.0.98.62:34489)
0928 23:12:01.083571 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.083573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.083656 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:12:01.083726 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.083727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.083733 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.083778 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.084337 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.084338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.084339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.084343 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.084359 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.084364 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.084410 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.084455 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.084469 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:12:01.086000 4999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:01.087469 4999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:01.090859 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43557 (local address 127.0.0.1:39700)
0928 23:12:01.089651 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.089692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.089744 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.089931 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.089935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.089940 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.090356 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.090356 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.090373 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.090396 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.090718 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.090724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.090727 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.090781 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.090793 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260928 23:12:01.091097 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.0.98.1:43557)
0928 23:12:01.089736 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.089740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.089752 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:12:01.089808 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.089815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.089826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.089896 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.090484 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.090490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.090490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.090494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.090530 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.090557 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.090609 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.090995 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.091012 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260928 23:12:01.091125 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.2:36847)
0928 23:12:01.089713 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.089717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.089879 (+ 162us) server_negotiation.cc:407] Connection header received
0928 23:12:01.089910 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.089913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.089923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.090420 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.090710 (+ 290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.090714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.090715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.090718 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.090748 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.090764 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.090818 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.091028 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.091046 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260928 23:12:01.090971 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36847 (local address 127.0.0.1:40648)
0928 23:12:01.089806 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.089814 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.089863 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.090014 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.090015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.090018 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.090640 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.090640 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.090646 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.090658 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.090816 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.090817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.090817 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.090872 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.090884 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260928 23:12:01.091302 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.0.98.3:33113)
0928 23:12:01.089968 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.089971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.089980 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:01.090001 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.090003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.090010 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.090071 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.090882 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.090889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.090889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.090893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.090924 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.090957 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.091010 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.091103 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.091117 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":67,"threads_started":1}
I20260928 23:12:01.091150 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.089406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33113 (local address 127.0.0.1:37480)
0928 23:12:01.089829 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.089837 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.089870 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.090344 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.090345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.090347 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.090791 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.090791 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.090799 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.090812 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.091044 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.091045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.091045 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.091088 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.091100 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":91,"threads_started":1}
I20260928 23:12:01.091977 5252 tablet_service.cc:1467] Processing CreateTablet for tablet 5758ad148de24bb596257dcc0883ec71 (DEFAULT_TABLE table=test-table [id=40e87440e3184ec9bdcff48c406004e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:01.091995 5121 tablet_service.cc:1467] Processing CreateTablet for tablet 5758ad148de24bb596257dcc0883ec71 (DEFAULT_TABLE table=test-table [id=40e87440e3184ec9bdcff48c406004e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:01.092321 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5758ad148de24bb596257dcc0883ec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.092324 5121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5758ad148de24bb596257dcc0883ec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.093633 5383 tablet_service.cc:1467] Processing CreateTablet for tablet 5758ad148de24bb596257dcc0883ec71 (DEFAULT_TABLE table=test-table [id=40e87440e3184ec9bdcff48c406004e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:01.093935 5383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5758ad148de24bb596257dcc0883ec71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.094887 5463 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Bootstrap starting.
I20260928 23:12:01.095058 5464 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Bootstrap starting.
I20260928 23:12:01.095687 5464 tablet_bootstrap.cc:654] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.095687 5463 tablet_bootstrap.cc:654] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.095985 5463 log.cc:824] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.095984 5464 log.cc:824] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.096472 5467 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Bootstrap starting.
I20260928 23:12:01.096782 5464 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: No bootstrap required, opened a new log
I20260928 23:12:01.096851 5464 ts_tablet_manager.cc:1397] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:01.097054 5463 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: No bootstrap required, opened a new log
I20260928 23:12:01.097119 5463 ts_tablet_manager.cc:1397] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:01.099357 5464 raft_consensus.cc:348] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.099530 5464 raft_consensus.cc:374] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.099483 5463 raft_consensus.cc:348] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.099562 5464 raft_consensus.cc:729] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a210c45165f04ac8ac96770542791910, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.099601 5463 raft_consensus.cc:374] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.099630 5463 raft_consensus.cc:729] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e7249eb69f47ff8ad0cc3853e2b211, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.099726 5463 consensus_queue.cc:260] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.099727 5464 consensus_queue.cc:260] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.099983 5463 ts_tablet_manager.cc:1428] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:12:01.099983 5464 ts_tablet_manager.cc:1428] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:12:01.100008 5187 heartbeater.cc:503] Master 127.0.98.62:34489 was elected leader, sending a full tablet report...
I20260928 23:12:01.100101 5318 heartbeater.cc:503] Master 127.0.98.62:34489 was elected leader, sending a full tablet report...
I20260928 23:12:01.100944 5467 tablet_bootstrap.cc:654] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.101263 5467 log.cc:824] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.101841 5467 tablet_bootstrap.cc:492] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: No bootstrap required, opened a new log
I20260928 23:12:01.101897 5467 ts_tablet_manager.cc:1397] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 23:12:01.103613 5467 raft_consensus.cc:348] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.103744 5467 raft_consensus.cc:374] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.103775 5467 raft_consensus.cc:729] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e0a703c7714061818f62e2a02f12e9, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.103884 5467 consensus_queue.cc:260] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.104108 5467 ts_tablet_manager.cc:1428] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:12:01.104147 5449 heartbeater.cc:503] Master 127.0.98.62:34489 was elected leader, sending a full tablet report...
W20260928 23:12:01.109380 5188 tablet.cc:2369] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:12:01.167160 5469 raft_consensus.cc:482] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:01.167297 5469 raft_consensus.cc:504] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.167689 5469 leader_election.cc:288] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45e7249eb69f47ff8ad0cc3853e2b211 (127.0.98.1:43557), 76e0a703c7714061818f62e2a02f12e9 (127.0.98.3:33113)
I20260928 23:12:01.168915 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.167882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33113 (local address 127.0.98.2:41871)
0928 23:12:01.168081 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.168091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.168134 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.168221 (+ 87us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.168222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.168224 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.168633 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.168633 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.168641 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.168654 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.168809 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.168810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.168811 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.168846 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.168857 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":151,"threads_started":1}
I20260928 23:12:01.168939 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.168031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41871 (local address 127.0.98.3:33113)
0928 23:12:01.168144 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.168146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.168152 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:01.168172 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.168173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.168177 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.168215 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.168729 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.168730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.168730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.168734 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.168752 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.168756 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.168813 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.168848 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.168861 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 23:12:01.169061 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.167855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43557 (local address 127.0.98.2:59749)
0928 23:12:01.168024 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.168036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.168077 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.168440 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.168441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.168443 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:01.168775 (+ 332us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:01.168775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:01.168782 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:01.168793 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:01.168956 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:01.168957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:01.168958 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 23:12:01.168994 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 23:12:01.169005 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 23:12:01.169296 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.168276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59749 (local address 127.0.98.1:43557)
0928 23:12:01.168334 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.168336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.168343 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:01.168365 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.168365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.168369 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.168411 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.168870 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.168872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.168872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.168876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.168890 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.168895 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.168936 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.169197 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.169232 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:12:01.169350 5403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5758ad148de24bb596257dcc0883ec71" candidate_uuid: "a210c45165f04ac8ac96770542791910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e0a703c7714061818f62e2a02f12e9" is_pre_election: true
I20260928 23:12:01.169497 5403 raft_consensus.cc:2454] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a210c45165f04ac8ac96770542791910 in term 0.
I20260928 23:12:01.169559 5141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5758ad148de24bb596257dcc0883ec71" candidate_uuid: "a210c45165f04ac8ac96770542791910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" is_pre_election: true
I20260928 23:12:01.169703 5141 raft_consensus.cc:2454] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a210c45165f04ac8ac96770542791910 in term 0.
I20260928 23:12:01.169704 5208 leader_election.cc:302] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e0a703c7714061818f62e2a02f12e9, a210c45165f04ac8ac96770542791910; no voters:
I20260928 23:12:01.169813 5469 raft_consensus.cc:2781] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:01.169876 5469 raft_consensus.cc:482] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:01.169898 5469 raft_consensus.cc:3037] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.170620 5469 raft_consensus.cc:504] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.170740 5469 leader_election.cc:288] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [CANDIDATE]: Term 1 election: Requested vote from peers 45e7249eb69f47ff8ad0cc3853e2b211 (127.0.98.1:43557), 76e0a703c7714061818f62e2a02f12e9 (127.0.98.3:33113)
I20260928 23:12:01.170874 5403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5758ad148de24bb596257dcc0883ec71" candidate_uuid: "a210c45165f04ac8ac96770542791910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76e0a703c7714061818f62e2a02f12e9"
I20260928 23:12:01.170908 5141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5758ad148de24bb596257dcc0883ec71" candidate_uuid: "a210c45165f04ac8ac96770542791910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45e7249eb69f47ff8ad0cc3853e2b211"
I20260928 23:12:01.170956 5403 raft_consensus.cc:3037] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.170965 5141 raft_consensus.cc:3037] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.171764 5403 raft_consensus.cc:2454] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a210c45165f04ac8ac96770542791910 in term 1.
I20260928 23:12:01.171764 5141 raft_consensus.cc:2454] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a210c45165f04ac8ac96770542791910 in term 1.
I20260928 23:12:01.171950 5208 leader_election.cc:302] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76e0a703c7714061818f62e2a02f12e9, a210c45165f04ac8ac96770542791910; no voters:
I20260928 23:12:01.172029 5469 raft_consensus.cc:2781] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:01.172174 5469 raft_consensus.cc:686] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 1 LEADER]: Becoming Leader. State: Replica: a210c45165f04ac8ac96770542791910, State: Running, Role: LEADER
I20260928 23:12:01.172256 5469 consensus_queue.cc:237] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } }
I20260928 23:12:01.173022 4999 catalog_manager.cc:5495] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 reported cstate change: term changed from 0 to 1, leader changed from <none> to a210c45165f04ac8ac96770542791910 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a210c45165f04ac8ac96770542791910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a210c45165f04ac8ac96770542791910" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 } health_report { overall_health: UNKNOWN } } }
I20260928 23:12:01.182433 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.181561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.0.98.2:36847)
0928 23:12:01.181607 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.181609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.181616 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:01.181675 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.181676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.181686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.181728 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.182208 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.182209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.182209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.182212 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.182224 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.182227 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.182289 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.182341 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.182351 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:12:01.183979 5141 raft_consensus.cc:1261] T 5758ad148de24bb596257dcc0883ec71 P 45e7249eb69f47ff8ad0cc3853e2b211 [term 1 FOLLOWER]: Refusing update from remote peer a210c45165f04ac8ac96770542791910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:01.183974 5403 raft_consensus.cc:1261] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 1 FOLLOWER]: Refusing update from remote peer a210c45165f04ac8ac96770542791910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:01.184280 5469 consensus_queue.cc:1035] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45e7249eb69f47ff8ad0cc3853e2b211" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:01.184405 5473 consensus_queue.cc:1035] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76e0a703c7714061818f62e2a02f12e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:01.187088 5478 mvcc.cc:205] Tried to move back new op lower bound from 7334449648367239168 to 7334449648322084864. Current Snapshot: MvccSnapshot[applied={T|T < 7334449648367239168 or (T in {7334449648367239168})}]
I20260928 23:12:01.187143 5477 mvcc.cc:205] Tried to move back new op lower bound from 7334449648367239168 to 7334449648322084864. Current Snapshot: MvccSnapshot[applied={T|T < 7334449648367239168}]
I20260928 23:12:01.189184 4999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:01.189294 4999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:01.189951 4999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df89611cd6b644a0b5f0dbb3f7c0086e: Sending DeleteTablet for 3 replicas of tablet 5758ad148de24bb596257dcc0883ec71
I20260928 23:12:01.190244 5252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5758ad148de24bb596257dcc0883ec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:01 UTC) from {username='slave'} at 127.0.0.1:40648
I20260928 23:12:01.190268 5383 tablet_service.cc:1514] Processing DeleteTablet for tablet 5758ad148de24bb596257dcc0883ec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:01 UTC) from {username='slave'} at 127.0.0.1:37480
I20260928 23:12:01.190260 5121 tablet_service.cc:1514] Processing DeleteTablet for tablet 5758ad148de24bb596257dcc0883ec71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:01 UTC) from {username='slave'} at 127.0.0.1:39700
I20260928 23:12:01.190480 5487 tablet_replica.cc:330] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: stopping tablet replica
I20260928 23:12:01.190593 5487 raft_consensus.cc:2229] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:01.190676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5059
I20260928 23:12:01.191340 5487 raft_consensus.cc:2258] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:01.191588 5486 tablet_replica.cc:330] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: stopping tablet replica
I20260928 23:12:01.191684 5486 raft_consensus.cc:2229] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:01.191778 5486 raft_consensus.cc:2258] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:01.191946 5487 ts_tablet_manager.cc:1905] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:01.192134 5486 ts_tablet_manager.cc:1905] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:01.193630 5487 ts_tablet_manager.cc:1918] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:01.193693 5487 log.cc:1196] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/5758ad148de24bb596257dcc0883ec71
I20260928 23:12:01.193813 5486 ts_tablet_manager.cc:1918] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:01.193939 5486 log.cc:1196] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/5758ad148de24bb596257dcc0883ec71
I20260928 23:12:01.193955 5487 ts_tablet_manager.cc:1939] T 5758ad148de24bb596257dcc0883ec71 P a210c45165f04ac8ac96770542791910: Deleting consensus metadata
I20260928 23:12:01.194252 5486 ts_tablet_manager.cc:1939] T 5758ad148de24bb596257dcc0883ec71 P 76e0a703c7714061818f62e2a02f12e9: Deleting consensus metadata
I20260928 23:12:01.194630 4986 catalog_manager.cc:4841] TS 76e0a703c7714061818f62e2a02f12e9 (127.0.98.3:33113): tablet 5758ad148de24bb596257dcc0883ec71 (table test-table [id=40e87440e3184ec9bdcff48c406004e2]) successfully deleted
W20260928 23:12:01.195861 4983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43557 (error 108)
I20260928 23:12:01.196291 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.196191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43557 (local address 127.0.0.1:39708)
0928 23:12:01.196232 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.196262 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:12:01.196395 4983 catalog_manager.cc:4584] TS 45e7249eb69f47ff8ad0cc3853e2b211 (127.0.98.1:43557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5758ad148de24bb596257dcc0883ec71: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43557: connect: Connection refused (error 111)
I20260928 23:12:01.197088 4985 catalog_manager.cc:4841] TS a210c45165f04ac8ac96770542791910 (127.0.98.2:36847): tablet 5758ad148de24bb596257dcc0883ec71 (table test-table [id=40e87440e3184ec9bdcff48c406004e2]) successfully deleted
I20260928 23:12:01.197394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5190
I20260928 23:12:01.202435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5321
I20260928 23:12:01.206579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 4968
2026-09-28T23:12:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (614 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 23:12:01.224439 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:12:01.225476 392 webserver.cc:460] Webserver started at https://127.0.0.1:36021/ using document root <none> and password file <none>
I20260928 23:12:01.225566 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:12:01.225718 392 webserver.cc:460] Webserver started at http://127.0.0.1:34467/ using document root <none> and password file <none>
2026-09-28T23:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:01Z Disabled control of system clock
I20260928 23:12:01.232430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36021/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/testchainca.cert with env {}
W20260928 23:12:01.309533 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:01.309734 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:01.309755 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:01.310876 5495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:01.310917 5495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:01.310930 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:01.310941 5495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:01.310952 5495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:12:01.310966 5495 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:12:01.310976 5495 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36021/jwks.json
I20260928 23:12:01.312443 5495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36021/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:01.312638 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:01.312803 5495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:01.317304 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.317356 5495 server_base.cc:1034] running on GCE node
W20260928 23:12:01.317324 5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.317288 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.317730 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:01.317996 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:01.319128 5495 hybrid_clock.cc:648] HybridClock initialized: now 1790637121319115 us; error 28 us; skew 500 ppm
I20260928 23:12:01.320238 5495 webserver.cc:460] Webserver started at http://127.0.98.62:33101/ using document root <none> and password file <none>
I20260928 23:12:01.320461 5495 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:01.320506 5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:01.320626 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:01.321681 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "898597a5bd1245efb7d7b773dd2417fc"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "e718ad4089d3192587bed7b40a787547"
server_key_iv: "2482c8bfe9bef43193ce08c57b5973fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.322005 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "898597a5bd1245efb7d7b773dd2417fc"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "e718ad4089d3192587bed7b40a787547"
server_key_iv: "2482c8bfe9bef43193ce08c57b5973fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.323155 5495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:01.323794 5509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.323976 5495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:01.324055 5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "898597a5bd1245efb7d7b773dd2417fc"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "e718ad4089d3192587bed7b40a787547"
server_key_iv: "2482c8bfe9bef43193ce08c57b5973fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.324115 5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:01.335887 5495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:01.336175 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:01.336298 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:01.339872 5495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42043
I20260928 23:12:01.339903 5561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42043 every 8 connection(s)
I20260928 23:12:01.340216 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:01.341090 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.343309 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: Bootstrap starting.
I20260928 23:12:01.343891 5562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.344152 5562 log.cc:824] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.344797 5562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: No bootstrap required, opened a new log
I20260928 23:12:01.346266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5495
I20260928 23:12:01.346359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:01.346374 5562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260928 23:12:01.346531 392 external_mini_cluster.cc:1428] Setting key cd32876aa3f9330fad94fd9e20525f6d
I20260928 23:12:01.346532 5562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.346570 5562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898597a5bd1245efb7d7b773dd2417fc, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.346719 5562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260928 23:12:01.346794 5562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:01.346828 5562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:01.346876 5562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.347446 5562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260928 23:12:01.347545 5562 leader_election.cc:302] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898597a5bd1245efb7d7b773dd2417fc; no voters:
I20260928 23:12:01.347735 5562 leader_election.cc:288] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:01.347756 5567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:01.347873 5567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [term 1 LEADER]: Becoming Leader. State: Replica: 898597a5bd1245efb7d7b773dd2417fc, State: Running, Role: LEADER
I20260928 23:12:01.347967 5567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } }
I20260928 23:12:01.347992 5562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:01.349326 5569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "898597a5bd1245efb7d7b773dd2417fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260928 23:12:01.349404 5569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.349453 5568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898597a5bd1245efb7d7b773dd2417fc. Latest consensus state: current_term: 1 leader_uuid: "898597a5bd1245efb7d7b773dd2417fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898597a5bd1245efb7d7b773dd2417fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42043 } } }
I20260928 23:12:01.349733 5568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc [sys.catalog]: This master's current role is: LEADER
W20260928 23:12:01.349998 5581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:12:01.350051 5581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:12:01.350100 5582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:01.350242 5582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:01.353000 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.347115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.0.98.62:42043)
0928 23:12:01.347310 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.347317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.347341 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:12:01.347433 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.347439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.347497 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.347639 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.347889 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.349723 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.350794 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.351393 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.351508 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.352333 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.352336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.352337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.352524 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.352567 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.352581 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.352638 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.352706 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.352781 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260928 23:12:01.353405 5582 catalog_manager.cc:1347] Generated new cluster ID: 926e918d3feb4d1fb8e34771a3b4a798
I20260928 23:12:01.353459 5582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:01.353876 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:12:01.367446 5582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:01.368731 5582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:01.372208 5582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 898597a5bd1245efb7d7b773dd2417fc: Generated new TSK 0
I20260928 23:12:01.372385 5582 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:12:01.400903 5592 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 23:12:01.401000 5592 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:12:01.401342 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.385019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.0.98.62:42043)
0928 23:12:01.385083 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.385085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.385203 (+ 118us) server_negotiation.cc:407] Connection header received
0928 23:12:01.385328 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.385329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.385343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.385408 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:12:01.385536 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.386317 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.387083 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.387403 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.387466 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.387618 (+ 152us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:12:01.401049 (+ 13431us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:12:01.401270 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 23:12:01.401405 5566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60386: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 23:12:01.401652 5591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:01.384949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42043 (local address 127.0.0.1:60386)
0928 23:12:01.385141 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.385150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.385226 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.385411 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.385412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.385424 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:12:01.385480 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.385482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.386397 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.386398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.387008 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.387009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.387463 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.387465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.387508 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.387526 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:12:01.401237 (+ 13711us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 23:12:01.401343 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36021/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 23:12:01.402136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5495
2026-09-28T23:12:01Z chronyd exiting
I20260928 23:12:01.437500 392 test_util.cc:175] -----------------------------------------------
I20260928 23:12:01.437582 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790637099107577-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 23:12:01.438269 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:12:01.438901 392 webserver.cc:460] Webserver started at https://127.0.0.1:45107/ using document root <none> and password file <none>
I20260928 23:12:01.438942 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:12:01.439066 392 webserver.cc:460] Webserver started at http://127.0.0.1:34513/ using document root <none> and password file <none>
2026-09-28T23:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:01Z Disabled control of system clock
I20260928 23:12:01.445767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37011
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/testchainca.cert with env {}
W20260928 23:12:01.524313 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:01.524514 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:01.524535 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:01.525687 5598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:01.525730 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:01.525743 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:01.525753 5598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:01.525769 5598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:12:01.525785 5598 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:12:01.525796 5598 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45107/jwks.json
I20260928 23:12:01.527304 5598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:01.527503 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:01.527669 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:01.532228 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.532295 5598 server_base.cc:1034] running on GCE node
W20260928 23:12:01.532196 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.532212 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.532609 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:01.532869 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:01.534008 5598 hybrid_clock.cc:648] HybridClock initialized: now 1790637121533998 us; error 49 us; skew 500 ppm
I20260928 23:12:01.535207 5598 webserver.cc:460] Webserver started at http://127.0.98.62:41185/ using document root <none> and password file <none>
I20260928 23:12:01.535420 5598 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:01.535467 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:01.535569 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:01.536628 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "3b28a42b9d1243a8806982af453eff25"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "dbf58ff455d3cd33c3e735e34bebbe95"
server_key_iv: "9edb40bbf767cf4e4e96cd53019a71a4"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.536952 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b28a42b9d1243a8806982af453eff25"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "dbf58ff455d3cd33c3e735e34bebbe95"
server_key_iv: "9edb40bbf767cf4e4e96cd53019a71a4"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.538225 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:01.538887 5612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.539047 5598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:01.539130 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "3b28a42b9d1243a8806982af453eff25"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "dbf58ff455d3cd33c3e735e34bebbe95"
server_key_iv: "9edb40bbf767cf4e4e96cd53019a71a4"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.539184 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:01.546900 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:01.547160 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:01.547258 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:01.550726 5598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37011
I20260928 23:12:01.550758 5664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37011 every 8 connection(s)
I20260928 23:12:01.551065 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:01.551945 5665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.554239 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25: Bootstrap starting.
I20260928 23:12:01.554780 5665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.555023 5665 log.cc:824] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.555625 5665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25: No bootstrap required, opened a new log
I20260928 23:12:01.557163 5665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260928 23:12:01.557341 5665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.557377 5665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b28a42b9d1243a8806982af453eff25, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.557515 5665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260928 23:12:01.557590 5665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:01.557619 5665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:01.557667 5665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.558148 5665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260928 23:12:01.558241 5665 leader_election.cc:302] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b28a42b9d1243a8806982af453eff25; no voters:
I20260928 23:12:01.558390 5665 leader_election.cc:288] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:01.558422 5668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:01.558583 5668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [term 1 LEADER]: Becoming Leader. State: Replica: 3b28a42b9d1243a8806982af453eff25, State: Running, Role: LEADER
I20260928 23:12:01.558631 5665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:01.558692 5668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } }
I20260928 23:12:01.559020 5669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b28a42b9d1243a8806982af453eff25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260928 23:12:01.559139 5669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.559125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5598
I20260928 23:12:01.559224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:01.559423 392 external_mini_cluster.cc:1428] Setting key f1dfa5de7ff9e719e9cd1fc961c194bf
I20260928 23:12:01.559990 5677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:01.560114 5670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b28a42b9d1243a8806982af453eff25. Latest consensus state: current_term: 1 leader_uuid: "3b28a42b9d1243a8806982af453eff25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b28a42b9d1243a8806982af453eff25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37011 } } }
I20260928 23:12:01.560189 5670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.560488 5677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:01.562032 5677 catalog_manager.cc:1347] Generated new cluster ID: 749d70e681c64ec69c824c03a0e99f5f
I20260928 23:12:01.562084 5677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:01.565124 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.559867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.0.98.62:37011)
0928 23:12:01.560043 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.560053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.560088 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:12:01.560231 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.560238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.560351 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.560502 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.560698 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.561828 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.562872 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.563553 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.563681 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.564314 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.564318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.564319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.564535 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.564579 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.564602 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.564679 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.564746 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.564917 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 23:12:01.565984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:12:01.568765 5677 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:01.569203 5677 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:01.575222 5677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b28a42b9d1243a8806982af453eff25: Generated new TSK 0
I20260928 23:12:01.575385 5677 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 23:12:01.601465 5695 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 23:12:01.601557 5695 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 23:12:01.601827 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.579726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.0.98.62:37011)
0928 23:12:01.579765 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.579767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.579869 (+ 102us) server_negotiation.cc:407] Connection header received
0928 23:12:01.579942 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.579943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.579952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.580010 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:12:01.580138 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.580618 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.581779 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.581975 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.582019 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.582182 (+ 163us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:12:01.601613 (+ 19431us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:12:01.601771 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:12:01.601877 5685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 23:12:01.601962 5694 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:01.579665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37011 (local address 127.0.0.1:32822)
0928 23:12:01.579810 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.579820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.579878 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.580022 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.580023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.580025 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:12:01.580081 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.580083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.580696 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.580697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.581698 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.581701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.582031 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.582037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.582095 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.582101 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:12:01.601740 (+ 19639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 23:12:01.601860 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45107/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260928 23:12:01.602245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5598
2026-09-28T23:12:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 23:12:01.638540 392 mini_oidc.cc:152] Starting JWKS server
I20260928 23:12:01.639179 392 webserver.cc:460] Webserver started at https://127.0.0.1:41697/ using document root <none> and password file <none>
I20260928 23:12:01.639218 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 23:12:01.639334 392 webserver.cc:460] Webserver started at http://127.0.0.1:43231/ using document root <none> and password file <none>
2026-09-28T23:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:01Z Disabled control of system clock
I20260928 23:12:01.645898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35735
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41697/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:12:01.722353 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:01.722553 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:01.722584 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:01.723701 5701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:01.723739 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:01.723757 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:01.723771 5701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:01.723786 5701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:12:01.723809 5701 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:12:01.723827 5701 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41697/jwks.json
I20260928 23:12:01.725144 5701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41697/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:01.725385 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:01.725634 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:01.729903 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.729931 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.730007 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.730261 5701 server_base.cc:1034] running on GCE node
I20260928 23:12:01.730500 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:01.730759 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:01.731892 5701 hybrid_clock.cc:648] HybridClock initialized: now 1790637121731877 us; error 30 us; skew 500 ppm
I20260928 23:12:01.733031 5701 webserver.cc:460] Webserver started at http://127.0.98.62:36547/ using document root <none> and password file <none>
I20260928 23:12:01.733253 5701 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:01.733299 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:01.733418 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:01.734442 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "552d1641e5434f44bdcf286fc72ccf8f"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "683a63f345f49a43c7d62f4fe3e4c5bb"
server_key_iv: "a637fde18765dd97e58f542dd5adb1c8"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.734766 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "552d1641e5434f44bdcf286fc72ccf8f"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "683a63f345f49a43c7d62f4fe3e4c5bb"
server_key_iv: "a637fde18765dd97e58f542dd5adb1c8"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.735904 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:12:01.736538 5715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.736677 5701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:01.736760 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "552d1641e5434f44bdcf286fc72ccf8f"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "683a63f345f49a43c7d62f4fe3e4c5bb"
server_key_iv: "a637fde18765dd97e58f542dd5adb1c8"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.736814 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:01.749629 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:01.749897 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:01.750005 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:01.753636 5701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35735
I20260928 23:12:01.753670 5767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35735 every 8 connection(s)
I20260928 23:12:01.754009 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:01.754645 5768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.757134 5768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f: Bootstrap starting.
I20260928 23:12:01.757688 5768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.757934 5768 log.cc:824] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.758519 5768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f: No bootstrap required, opened a new log
I20260928 23:12:01.759608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5701
I20260928 23:12:01.759702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:01.759874 392 external_mini_cluster.cc:1428] Setting key 421049d96fdeb069edfc0565c9ceef91
I20260928 23:12:01.760351 5768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260928 23:12:01.760528 5768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.760561 5768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552d1641e5434f44bdcf286fc72ccf8f, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.760685 5768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260928 23:12:01.760744 5768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:01.760766 5768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:01.760809 5768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.761339 5768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260928 23:12:01.761421 5768 leader_election.cc:302] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 552d1641e5434f44bdcf286fc72ccf8f; no voters:
I20260928 23:12:01.761559 5768 leader_election.cc:288] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:01.761587 5773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:01.761739 5773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [term 1 LEADER]: Becoming Leader. State: Replica: 552d1641e5434f44bdcf286fc72ccf8f, State: Running, Role: LEADER
I20260928 23:12:01.761801 5768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:01.761870 5773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } }
I20260928 23:12:01.762166 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 552d1641e5434f44bdcf286fc72ccf8f. Latest consensus state: current_term: 1 leader_uuid: "552d1641e5434f44bdcf286fc72ccf8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260928 23:12:01.762374 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.762311 5775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "552d1641e5434f44bdcf286fc72ccf8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d1641e5434f44bdcf286fc72ccf8f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35735 } } }
I20260928 23:12:01.762773 5775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.762804 5782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:01.763546 5782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:01.765264 5782 catalog_manager.cc:1347] Generated new cluster ID: f27e3f8d18a74f92b5f9aaecf278ec59
I20260928 23:12:01.765355 5782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:01.765790 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.760356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.0.98.62:35735)
0928 23:12:01.760546 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.760555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.760587 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:12:01.760702 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.760709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.760776 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.760917 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.761093 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.762662 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.763700 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.764272 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.764412 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.765052 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.765056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.765056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.765277 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.765317 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.765332 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.765396 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.765510 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.765591 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260928 23:12:01.766707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:12:01.776687 5782 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:01.777405 5782 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:01.786273 5782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 552d1641e5434f44bdcf286fc72ccf8f: Generated new TSK 0
I20260928 23:12:01.786439 5782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:12:01.800022 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.796735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.98.62:35735)
0928 23:12:01.796774 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.796776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.796832 (+ 56us) server_negotiation.cc:407] Connection header received
0928 23:12:01.796912 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.796913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.796922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.796976 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:12:01.797093 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.797597 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.798756 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.798928 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.798966 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.799132 (+ 166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:12:01.799844 (+ 712us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:12:01.799959 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:12:01.800076 5772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 23:12:01.800112 5797 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:01.796628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35735 (local address 127.0.0.1:53528)
0928 23:12:01.796775 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:01.796786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:01.796850 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:01.796979 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:01.796980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:01.796982 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:12:01.797038 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.797040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.797676 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.797677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.798683 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:01.798687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.798988 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.798990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:01.799048 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.799054 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:12:01.799931 (+ 877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 23:12:01.800018 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35735: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41697/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 23:12:01.800513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5701
E20260928 23:12:01.812089 5798 webserver.cc:533] Webserver: sslize failed:
E20260928 23:12:01.812150 5798 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T23:12:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T23:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:01Z Disabled control of system clock
I20260928 23:12:01.844282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35199
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35199
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/testchainca.cert with env {}
W20260928 23:12:01.921541 5802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:01.921752 5802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:01.921772 5802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:01.922866 5802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:01.922899 5802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:01.922911 5802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:01.922921 5802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:01.922932 5802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 23:12:01.922948 5802 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 23:12:01.922963 5802 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 23:12:01.924341 5802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:01.924531 5802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:01.924705 5802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:01.929071 5808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.929102 5807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:01.929119 5810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:01.929291 5802 server_base.cc:1034] running on GCE node
I20260928 23:12:01.929517 5802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:01.929797 5802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:01.930938 5802 hybrid_clock.cc:648] HybridClock initialized: now 1790637121930929 us; error 47 us; skew 500 ppm
I20260928 23:12:01.932083 5802 webserver.cc:460] Webserver started at http://127.0.98.62:44685/ using document root <none> and password file <none>
I20260928 23:12:01.932279 5802 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:01.932322 5802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:01.932435 5802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:01.933490 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "0c3c90162fcb48cab982790e722381ae"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "005f2d730728e3252f4acce22950e7bf"
server_key_iv: "9ad2bcb0f65a4c981cb86746e0027854"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.933816 5802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c3c90162fcb48cab982790e722381ae"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "005f2d730728e3252f4acce22950e7bf"
server_key_iv: "9ad2bcb0f65a4c981cb86746e0027854"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.934973 5802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:01.935607 5816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:01.935760 5802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:01.935843 5802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "0c3c90162fcb48cab982790e722381ae"
format_stamp: "Formatted at 2026-09-28 23:12:01 on dist-test-slave-6vgd"
server_key: "005f2d730728e3252f4acce22950e7bf"
server_key_iv: "9ad2bcb0f65a4c981cb86746e0027854"
server_key_version: "encryptionkey@0"
I20260928 23:12:01.935897 5802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:01.961272 5802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:01.961577 5802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:01.961678 5802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:01.965512 5802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35199
I20260928 23:12:01.965570 5868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35199 every 8 connection(s)
I20260928 23:12:01.965876 5802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:01.966754 5869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:01.968068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5802
I20260928 23:12:01.968153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:01.968308 392 external_mini_cluster.cc:1428] Setting key 2a7507592d02c90f0560e6c8037acd95
I20260928 23:12:01.969300 5869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae: Bootstrap starting.
I20260928 23:12:01.969980 5869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:01.970287 5869 log.cc:824] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:01.970988 5869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae: No bootstrap required, opened a new log
I20260928 23:12:01.973057 5869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } }
I20260928 23:12:01.973297 5869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:01.973356 5869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3c90162fcb48cab982790e722381ae, State: Initialized, Role: FOLLOWER
I20260928 23:12:01.973475 5869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } }
I20260928 23:12:01.973558 5869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:01.973593 5869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:01.973636 5869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:01.974036 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:01.968790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.0.98.62:35199)
0928 23:12:01.968994 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:01.969003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:01.969037 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:12:01.969568 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:01.969578 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:01.969680 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:01.969829 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:01.970000 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.971279 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.972014 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:01.972631 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:01.972776 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:01.973367 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:01.973372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:01.973372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:01.973569 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:01.973611 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:01.973625 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:01.973691 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:01.973765 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:12:01.973844 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260928 23:12:01.974277 5869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } }
I20260928 23:12:01.974372 5869 leader_election.cc:302] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c3c90162fcb48cab982790e722381ae; no voters:
I20260928 23:12:01.974519 5869 leader_election.cc:288] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:01.974552 5874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:01.974840 5874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [term 1 LEADER]: Becoming Leader. State: Replica: 0c3c90162fcb48cab982790e722381ae, State: Running, Role: LEADER
I20260928 23:12:01.974920 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:12:01.974880 5869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:01.975517 5874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } }
I20260928 23:12:01.976058 5875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c3c90162fcb48cab982790e722381ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } } }
I20260928 23:12:01.976140 5875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.976368 5876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c3c90162fcb48cab982790e722381ae. Latest consensus state: current_term: 1 leader_uuid: "0c3c90162fcb48cab982790e722381ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c3c90162fcb48cab982790e722381ae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35199 } } }
I20260928 23:12:01.976432 5876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:01.976464 5883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:01.976956 5883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:01.978492 5883 catalog_manager.cc:1347] Generated new cluster ID: 3120f71f498f423f842b6eadd7f8554a
I20260928 23:12:01.978536 5883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:02.003868 5883 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:02.004417 5883 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:02.006882 5883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c3c90162fcb48cab982790e722381ae: Generated new TSK 0
I20260928 23:12:02.007052 5883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:12:02.089104 5872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:02.020180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.0.98.62:35199)
0928 23:12:02.020234 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:02.020237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:02.020319 (+ 82us) server_negotiation.cc:407] Connection header received
0928 23:12:02.020377 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:02.020378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:02.020391 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:02.020464 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 23:12:02.020573 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:02.021379 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:02.022151 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:02.022461 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:02.022525 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:02.022674 (+ 149us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 23:12:02.088833 (+ 66159us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:12:02.089021 (+ 188us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50490: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 23:12:02.089182 5872 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50490: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 23:12:02.089265 5898 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:02.020117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35199 (local address 127.0.0.1:50490)
0928 23:12:02.020257 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:02.020267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:02.020324 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:02.020470 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:02.020471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:02.020475 (+ 4us) client_negotiation.cc:189] Negotiated authn=JWT
0928 23:12:02.020525 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:02.020527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:02.021478 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:02.021479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:02.022082 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:02.022084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:02.022528 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:02.022532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:02.022576 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:02.022582 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 23:12:02.088992 (+ 66410us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 23:12:02.089134 (+ 142us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260928 23:12:02.089607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5802
2026-09-28T23:12:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (268 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:02 dist-test-slave-6vgd krb5kdc[5910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:02 dist-test-slave-6vgd krb5kdc[5910](info): set up 2 sockets
Sep 28 23:12:02 dist-test-slave-6vgd krb5kdc[5910](info): commencing operation
krb5kdc: starting...
W20260928 23:12:03.209132 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:12:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:12:03.317494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:03.395138 5923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:03.395335 5923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:03.395356 5923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:03.396435 5923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:03.396466 5923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:03.396479 5923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:03.396490 5923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:03.396507 5923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:03.397976 5923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:03.398164 5923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:03.398335 5923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:03.402731 5931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:03.402757 5929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.402858 5923 server_base.cc:1034] running on GCE node
W20260928 23:12:03.402733 5928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.403227 5923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:03.403497 5923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:03.404632 5923 hybrid_clock.cc:648] HybridClock initialized: now 1790637123404618 us; error 29 us; skew 500 ppm
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.406935 5923 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:12:03.407325 5923 webserver.cc:460] Webserver started at http://127.0.98.62:40463/ using document root <none> and password file <none>
I20260928 23:12:03.407506 5923 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:03.407552 5923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:03.407670 5923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:03.408663 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "4649a9d3a7e7454c801d3c5c383ad459"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "6aa5ff9ca5f8a017fefb1491db55336e"
server_key_iv: "cd97ef006823621186cc89557a825bd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.408988 5923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "4649a9d3a7e7454c801d3c5c383ad459"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "6aa5ff9ca5f8a017fefb1491db55336e"
server_key_iv: "cd97ef006823621186cc89557a825bd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.410198 5923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:03.410822 5938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.410970 5923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:03.411053 5923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "4649a9d3a7e7454c801d3c5c383ad459"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "6aa5ff9ca5f8a017fefb1491db55336e"
server_key_iv: "cd97ef006823621186cc89557a825bd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.411118 5923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:03.446596 5923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:03.448766 5923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:03.448909 5923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:03.452587 5923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34457
I20260928 23:12:03.452637 5990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34457 every 8 connection(s)
I20260928 23:12:03.453104 5923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:03.453771 5991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:03.456020 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: Bootstrap starting.
I20260928 23:12:03.456561 5991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:03.456822 5991 log.cc:824] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:03.457443 5991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: No bootstrap required, opened a new log
I20260928 23:12:03.458933 5991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } }
I20260928 23:12:03.459069 5991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:03.459090 5991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4649a9d3a7e7454c801d3c5c383ad459, State: Initialized, Role: FOLLOWER
I20260928 23:12:03.459223 5991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } }
I20260928 23:12:03.459291 5991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:03.459313 5991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:03.459357 5991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:03.460175 5991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } }
I20260928 23:12:03.460264 5991 leader_election.cc:302] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4649a9d3a7e7454c801d3c5c383ad459; no voters:
I20260928 23:12:03.460419 5991 leader_election.cc:288] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:03.460459 5994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:03.460609 5994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [term 1 LEADER]: Becoming Leader. State: Replica: 4649a9d3a7e7454c801d3c5c383ad459, State: Running, Role: LEADER
I20260928 23:12:03.460645 5991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:03.460692 5994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } }
I20260928 23:12:03.461007 5995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4649a9d3a7e7454c801d3c5c383ad459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } } }
I20260928 23:12:03.461097 5995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:03.461012 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4649a9d3a7e7454c801d3c5c383ad459. Latest consensus state: current_term: 1 leader_uuid: "4649a9d3a7e7454c801d3c5c383ad459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4649a9d3a7e7454c801d3c5c383ad459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34457 } } }
I20260928 23:12:03.461241 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:03.461642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 5923
I20260928 23:12:03.461738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:03.461935 392 external_mini_cluster.cc:1428] Setting key 408fd5b68fd28a3dd4d13ebbf17f1944
W20260928 23:12:03.462364 6010 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:12:03.462500 6010 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:12:03.462579 6002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:03.462756 6002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:03.465693 6002 catalog_manager.cc:1347] Generated new cluster ID: 55588fbdc4de4746942ee61a7545346a
I20260928 23:12:03.465744 6002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:03.472669 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.462365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.98.62:34457)
0928 23:12:03.462765 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.462774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.462803 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:12:03.462917 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.462924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.462977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.463106 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.463724 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.465119 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.465968 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.466598 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.466708 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.469351 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.469355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.469355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.469367 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.471168 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.471546 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.471548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.471549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.471642 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.471829 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.471829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.471830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.472093 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.472226 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.472320 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.472441 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":77,"threads_started":1}
I20260928 23:12:03.491998 6002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:03.492748 6002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:03.497073 6002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4649a9d3a7e7454c801d3c5c383ad459: Generated new TSK 0
I20260928 23:12:03.497279 6002 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:12:03.520520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:03.600186 6017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:03.600389 6017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:03.600418 6017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:03.601591 6017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:03.601631 6017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:03.601666 6017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:03.603106 6017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:03.603330 6017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:03.603530 6017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:03.608069 6025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:03.608074 6022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.608268 6017 server_base.cc:1034] running on GCE node
W20260928 23:12:03.608389 6023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.608592 6017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:03.608829 6017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:03.609963 6017 hybrid_clock.cc:648] HybridClock initialized: now 1790637123609948 us; error 27 us; skew 500 ppm
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.612205 6017 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:12:03.612617 6017 webserver.cc:460] Webserver started at http://127.0.98.1:38157/ using document root <none> and password file <none>
I20260928 23:12:03.612803 6017 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:03.612847 6017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:03.612962 6017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:03.614027 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cd0b97642ec40599d0c7eea520113e6"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "cb3bd92b79e9f23180871298f6f264b6"
server_key_iv: "d95d2fabaadbb85b21bfbfb293bb320d"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.614334 6017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cd0b97642ec40599d0c7eea520113e6"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "cb3bd92b79e9f23180871298f6f264b6"
server_key_iv: "d95d2fabaadbb85b21bfbfb293bb320d"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.615512 6017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:12:03.616115 6032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.616238 6017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:03.616315 6017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "6cd0b97642ec40599d0c7eea520113e6"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "cb3bd92b79e9f23180871298f6f264b6"
server_key_iv: "d95d2fabaadbb85b21bfbfb293bb320d"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.616379 6017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:03.634249 6017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:03.635958 6017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:03.636089 6017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:03.636289 6017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:03.636574 6017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:03.636606 6017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.636636 6017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:03.636658 6017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.641911 6017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36605
I20260928 23:12:03.641959 6145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36605 every 8 connection(s)
I20260928 23:12:03.642243 6017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:03.644892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6017
I20260928 23:12:03.644975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:03.645155 392 external_mini_cluster.cc:1428] Setting key e111f30153c3d81baaad38b2dcd84e9c
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:03.651607 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.642859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51447 (local address 127.0.98.62:34457)
0928 23:12:03.642916 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.642918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.643119 (+ 201us) server_negotiation.cc:407] Connection header received
0928 23:12:03.644003 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.644004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.644015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.644064 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.644619 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.645145 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.646488 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.646688 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.646741 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.648865 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.648866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.648867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.648878 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.650365 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.650721 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.650722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.650723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.650783 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.650994 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.650995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.650995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.651127 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.651187 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.651422 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.651510 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:12:03.651661 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.642856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34457 (local address 127.0.98.1:51447)
0928 23:12:03.643002 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:03.643038 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:03.643919 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:03.644120 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:03.644123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:03.644370 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:03.644556 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.644560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.645256 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.645257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.646407 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.646410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.646762 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.646763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.646898 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.647436 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:03.647444 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:03.648748 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:03.650439 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.650445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.650447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.650653 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.650849 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.650850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.650850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.650939 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.651185 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:03.651188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:03.651298 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 23:12:03.651380 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:12:03.651500 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260928 23:12:03.652323 6146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34457
I20260928 23:12:03.652419 6146 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:03.652590 6146 heartbeater.cc:511] Master 127.0.98.62:34457 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:12:03.653162 5955 ts_manager.cc:194] Registered new tserver with Master: 6cd0b97642ec40599d0c7eea520113e6 (127.0.98.1:36605)
I20260928 23:12:03.653834 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:12:03.668099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:03.746585 6151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:03.746785 6151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:03.746824 6151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:03.747958 6151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:03.748000 6151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:03.748034 6151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:03.749496 6151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:03.749717 6151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:03.749924 6151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:03.754679 6157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.754796 6151 server_base.cc:1034] running on GCE node
W20260928 23:12:03.754673 6156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:03.754678 6159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.755110 6151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:03.755334 6151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:03.756469 6151 hybrid_clock.cc:648] HybridClock initialized: now 1790637123756459 us; error 27 us; skew 500 ppm
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.758810 6151 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:12:03.759178 6151 webserver.cc:460] Webserver started at http://127.0.98.2:45115/ using document root <none> and password file <none>
I20260928 23:12:03.759367 6151 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:03.759406 6151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:03.759498 6151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:03.760497 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "ec9a6a935472471e92fca61bfc4be259"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "e865ab74289fa243d16c8b997b7070ae"
server_key_iv: "8bc18d9d8b3e23fd0e06eb00be377fc7"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.760780 6151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ec9a6a935472471e92fca61bfc4be259"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "e865ab74289fa243d16c8b997b7070ae"
server_key_iv: "8bc18d9d8b3e23fd0e06eb00be377fc7"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.762020 6151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:12:03.762677 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.762800 6151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.762869 6151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "ec9a6a935472471e92fca61bfc4be259"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "e865ab74289fa243d16c8b997b7070ae"
server_key_iv: "8bc18d9d8b3e23fd0e06eb00be377fc7"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.762921 6151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:03.780490 6151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:03.782372 6151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:03.782505 6151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:03.782737 6151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:03.783063 6151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:03.783097 6151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.783119 6151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:03.783142 6151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.788578 6151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37411
I20260928 23:12:03.788642 6279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37411 every 8 connection(s)
I20260928 23:12:03.788915 6151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:03.792632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6151
I20260928 23:12:03.792716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:03.792873 392 external_mini_cluster.cc:1428] Setting key c24f815e02b58869fb46a1b3515a5a84
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:03.798193 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.789632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35729 (local address 127.0.98.62:34457)
0928 23:12:03.789695 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.789697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.789901 (+ 204us) server_negotiation.cc:407] Connection header received
0928 23:12:03.790789 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.790791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.790799 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.790861 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.791449 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.791948 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.792997 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.793281 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.793337 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.795749 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.795750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.795751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.795759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.796905 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.797287 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.797288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.797289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.797350 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.797576 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.797577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.797577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.797694 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.797750 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.798013 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.798104 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:12:03.798415 6282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.789597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34457 (local address 127.0.98.2:35729)
0928 23:12:03.789762 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:03.789804 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:03.790703 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:03.790925 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:03.790928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:03.791186 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:03.791381 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.791386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.792016 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.792016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.792923 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.792926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.793365 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.793367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.793499 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.793999 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:03.794011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:03.795653 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:03.796976 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.796984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.796986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.797203 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.797409 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.797410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.797410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.797520 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.797747 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:03.797754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:03.797900 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 23:12:03.797988 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:12:03.798143 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 23:12:03.798853 6280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34457
I20260928 23:12:03.798969 6280 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:03.799167 6280 heartbeater.cc:511] Master 127.0.98.62:34457 requested a full tablet report, sending...
I20260928 23:12:03.799546 5955 ts_manager.cc:194] Registered new tserver with Master: ec9a6a935472471e92fca61bfc4be259 (127.0.98.2:37411)
I20260928 23:12:03.799921 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35729
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:12:03.816143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:03.894789 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:03.894999 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:03.895020 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:03.896129 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:03.896164 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:03.896188 6285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:03.897622 6285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:03.897804 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:03.897987 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:03.902704 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:03.902704 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:03.902762 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:03.902838 6285 server_base.cc:1034] running on GCE node
I20260928 23:12:03.903119 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:03.903376 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:03.904527 6285 hybrid_clock.cc:648] HybridClock initialized: now 1790637123904513 us; error 24 us; skew 500 ppm
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.906804 6285 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:12:03.907222 6285 webserver.cc:460] Webserver started at http://127.0.98.3:42941/ using document root <none> and password file <none>
I20260928 23:12:03.907405 6285 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:03.907444 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:03.907532 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:03.908504 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a9000b1ec6547fba5e1832aaccfc035"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "2f7a9d61968709b7741f56550aad7d1b"
server_key_iv: "06824b718c91bf26a17af7cd2849adaf"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.908783 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a9000b1ec6547fba5e1832aaccfc035"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "2f7a9d61968709b7741f56550aad7d1b"
server_key_iv: "06824b718c91bf26a17af7cd2849adaf"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.909964 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:03.910576 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.910713 6285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:03.910812 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "6a9000b1ec6547fba5e1832aaccfc035"
format_stamp: "Formatted at 2026-09-28 23:12:03 on dist-test-slave-6vgd"
server_key: "2f7a9d61968709b7741f56550aad7d1b"
server_key_iv: "06824b718c91bf26a17af7cd2849adaf"
server_key_version: "encryptionkey@0"
I20260928 23:12:03.910892 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:03.926170 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:03.928192 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:03.928314 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:03.928516 6285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:03.928788 6285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:03.928818 6285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.928841 6285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:03.928864 6285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:03.934401 6285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34399
I20260928 23:12:03.934458 6413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34399 every 8 connection(s)
I20260928 23:12:03.934744 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:03.940423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6285
I20260928 23:12:03.940510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:03.940668 392 external_mini_cluster.cc:1428] Setting key 0550b74bbcad239d5e357c7f20875731
I20260928 23:12:03.943172 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.935359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44659 (local address 127.0.98.62:34457)
0928 23:12:03.935422 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.935423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.935617 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:12:03.936521 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.936523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.936532 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.936580 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.937132 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.937661 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.938596 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.938765 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.938807 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.940684 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.940685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.940686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.940693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.941987 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.942331 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.942333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.942333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.942394 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.942602 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.942603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.942604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.942710 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.942761 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.943015 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.943087 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 23:12:03.943331 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.935356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34457 (local address 127.0.98.3:44659)
0928 23:12:03.935514 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:03.935549 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:03.936435 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:03.936638 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:03.936641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:03.936876 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:03.937069 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.937074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.937733 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.937733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.938535 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:03.938536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.938815 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.938816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:03.938907 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.939294 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:03.939302 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:03.940600 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:03.942057 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.942063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.942065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.942267 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.942446 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:03.942447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:03.942447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:03.942551 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:03.942761 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:03.942766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:03.942902 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 23:12:03.942986 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:12:03.943132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260928 23:12:03.943691 6414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34457
I20260928 23:12:03.943769 6414 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:03.943908 6414 heartbeater.cc:511] Master 127.0.98.62:34457 requested a full tablet report, sending...
I20260928 23:12:03.944202 5955 ts_manager.cc:194] Registered new tserver with Master: 6a9000b1ec6547fba5e1832aaccfc035 (127.0.98.3:34399)
I20260928 23:12:03.944566 5955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44659
I20260928 23:12:03.952219 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/test-admin.keytab.
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.965072 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 23:12:03.966522 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:03.965864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.0.98.1:36605)
0928 23:12:03.966029 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.966033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.966044 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:12:03.966069 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.966071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.966085 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.966146 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.966451 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33032: BlockingRecv error: recv got EOF from 127.0.0.1:33032 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
W20260928 23:12:03.966619 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.967536 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.968264 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:12:03.975090 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.968712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.0.98.1:36605)
0928 23:12:03.968764 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.968766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.968813 (+ 47us) server_negotiation.cc:407] Connection header received
0928 23:12:03.968852 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.968853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.968863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.968913 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.969245 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.969964 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.970619 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.970929 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.970975 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.972785 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.972793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.972794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.972801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.974091 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.974385 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.974388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.974389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.974438 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.974598 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.974599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.974599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.974730 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.974874 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.974917 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.974990 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 23:12:03.976646 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:03.976195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.0.98.1:36605)
0928 23:12:03.976250 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.976252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.976260 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:03.976316 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.976317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.976327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.976375 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.976603 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33042: BlockingRecv error: recv got EOF from 127.0.0.1:33042 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 23:12:03.976693 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.977524 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.978209 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:12:03.984112 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.978654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.98.1:36605)
0928 23:12:03.978703 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.978705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.978712 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:03.978758 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.978758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.978767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.978810 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.979112 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.979595 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.980223 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.980420 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.980471 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.982146 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.982147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.982147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.982153 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.983205 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.983478 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.983479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.983480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.983529 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.983679 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.983680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.983680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.983771 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.983847 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.983951 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.984022 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 23:12:03.985359 6421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:03.984902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.0.98.1:36605)
0928 23:12:03.984949 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.984951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.984958 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:03.985001 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.985002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.985009 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.985043 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.985327 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33054: BlockingRecv error: recv got EOF from 127.0.0.1:33054 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 23:12:03.985442 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790637099107577-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.986268 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:03.987010 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 23:12:03 dist-test-slave-6vgd krb5kdc[5910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:12:03.992635 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:03.987419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.0.98.1:36605)
0928 23:12:03.987466 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:03.987468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:03.987497 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:12:03.987543 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:03.987544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:03.987551 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:03.987596 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:03.987878 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.988310 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.988901 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:03.989065 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:03.989105 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:03.990778 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:03.990779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:03.990780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:03.990786 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:03.991897 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.992131 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.992132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.992132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.992174 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:03.992287 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:03.992287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:03.992288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:03.992362 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:03.992435 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:03.992506 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 23:12:03.992555 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:12:03.993119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6017
I20260928 23:12:03.997488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6151
I20260928 23:12:04.001406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6285
I20260928 23:12:04.004988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 5923
2026-09-28T23:12:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1741 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:04 dist-test-slave-6vgd krb5kdc[6434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:04 dist-test-slave-6vgd krb5kdc[6434](info): set up 2 sockets
Sep 28 23:12:04 dist-test-slave-6vgd krb5kdc[6434](info): commencing operation
krb5kdc: starting...
W20260928 23:12:04.988312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 23:12:05.149896 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.161s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 23:12:05.270656 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.121s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 23:12:05.438442 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.168s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 23:12:05.489934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:05.566967 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:05.567178 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:05.567206 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:05.568331 6447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:05.568369 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:05.568387 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:05.568403 6447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:05.568419 6447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:05.569924 6447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:05.570161 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:05.570358 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:05.574846 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:05.574862 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:05.574859 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:05.574985 6447 server_base.cc:1034] running on GCE node
I20260928 23:12:05.575304 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:05.575568 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:05.576702 6447 hybrid_clock.cc:648] HybridClock initialized: now 1790637125576692 us; error 28 us; skew 500 ppm
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:05.584558 6447 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 23:12:05.585142 6447 webserver.cc:460] Webserver started at http://127.0.98.62:39473/ using document root <none> and password file <none>
I20260928 23:12:05.585412 6447 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:05.585475 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:05.585618 6447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:05.587103 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "81cb14f6bcb1458a834b9e4f35a277d3"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "3e1a523b294d84f34cd2b5cf2cabf3e5"
server_key_iv: "7bae40879b27b717be4871256bfc8dc6"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.587522 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "81cb14f6bcb1458a834b9e4f35a277d3"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "3e1a523b294d84f34cd2b5cf2cabf3e5"
server_key_iv: "7bae40879b27b717be4871256bfc8dc6"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.589138 6447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 23:12:05.589826 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.589973 6447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.590067 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "81cb14f6bcb1458a834b9e4f35a277d3"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "3e1a523b294d84f34cd2b5cf2cabf3e5"
server_key_iv: "7bae40879b27b717be4871256bfc8dc6"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.590134 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:05.598997 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:05.601189 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:05.601351 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:05.606292 6447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35539
I20260928 23:12:05.606323 6514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35539 every 8 connection(s)
I20260928 23:12:05.606664 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:05.607573 6515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:05.609850 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Bootstrap starting.
I20260928 23:12:05.610391 6515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:05.610634 6515 log.cc:824] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:05.611238 6515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: No bootstrap required, opened a new log
I20260928 23:12:05.612766 6515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260928 23:12:05.612921 6515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:05.612954 6515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81cb14f6bcb1458a834b9e4f35a277d3, State: Initialized, Role: FOLLOWER
I20260928 23:12:05.613101 6515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260928 23:12:05.613204 6515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:05.613266 6515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:05.613312 6515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:05.613840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6447
I20260928 23:12:05.613933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:05.614104 392 external_mini_cluster.cc:1428] Setting key 143078110367aed966f89fe50681d9cf
I20260928 23:12:05.614189 6515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260928 23:12:05.614293 6515 leader_election.cc:302] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81cb14f6bcb1458a834b9e4f35a277d3; no voters:
I20260928 23:12:05.614454 6515 leader_election.cc:288] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:05.614486 6518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:05.614674 6518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 LEADER]: Becoming Leader. State: Replica: 81cb14f6bcb1458a834b9e4f35a277d3, State: Running, Role: LEADER
I20260928 23:12:05.614781 6515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:05.614754 6518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } }
I20260928 23:12:05.615114 6522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } }
I20260928 23:12:05.615126 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81cb14f6bcb1458a834b9e4f35a277d3. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } }
I20260928 23:12:05.615324 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:05.615501 6522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:05.615757 6529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:05.616251 6529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:05.617658 6529 catalog_manager.cc:1347] Generated new cluster ID: a321c9f99cfe4dcab04c616119533bb2
I20260928 23:12:05.617707 6529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:05.626113 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:05.614528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.0.98.62:35539)
0928 23:12:05.614684 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:05.614694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:05.615403 (+ 709us) server_negotiation.cc:407] Connection header received
0928 23:12:05.615615 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:05.615624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:05.615685 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:05.615798 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:05.616842 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.618182 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.619202 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.619823 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.619936 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:05.622516 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:05.622521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:05.622521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:05.622535 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:05.624462 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.624815 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.624819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.624820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.624946 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.625128 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.625129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.625129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.625478 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:05.625634 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:05.625727 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:12:05.625867 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260928 23:12:05.656759 6529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:05.657570 6529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:05.665345 6529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Generated new TSK 0
I20260928 23:12:05.665547 6529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 23:12:05.696355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:05.773793 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:05.773983 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:05.774004 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:05.775072 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:05.775106 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:05.775130 6541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:05.776458 6541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:05.776638 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:05.776803 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:12:05.781567 6541 server_base.cc:1034] running on GCE node
W20260928 23:12:05.781491 6547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:05.781494 6549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:05.781522 6546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:05.781929 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:05.782168 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:05.783306 6541 hybrid_clock.cc:648] HybridClock initialized: now 1790637125783294 us; error 25 us; skew 500 ppm
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:05.785686 6541 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:12:05.786120 6541 webserver.cc:460] Webserver started at http://127.0.98.1:34845/ using document root <none> and password file <none>
I20260928 23:12:05.786316 6541 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:05.786363 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:05.786489 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:05.787539 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8d060f45c3d4d3bb655319ccf49d21f"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1aba2b1fcd680b2b7f774a2011f1e4d0"
server_key_iv: "efedf0d8c8be6d245ce8f2d29254eeb9"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.787878 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8d060f45c3d4d3bb655319ccf49d21f"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1aba2b1fcd680b2b7f774a2011f1e4d0"
server_key_iv: "efedf0d8c8be6d245ce8f2d29254eeb9"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.789057 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:12:05.789749 6556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.789897 6541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.789974 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "a8d060f45c3d4d3bb655319ccf49d21f"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1aba2b1fcd680b2b7f774a2011f1e4d0"
server_key_iv: "efedf0d8c8be6d245ce8f2d29254eeb9"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.790047 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:05.810088 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:05.811872 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:05.811985 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:05.812171 6541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:05.812410 6541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:05.812439 6541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.812459 6541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:05.812474 6541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.817948 6541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41953
I20260928 23:12:05.817998 6669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41953 every 8 connection(s)
I20260928 23:12:05.818289 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:05.820531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6541
I20260928 23:12:05.820622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:05.820775 392 external_mini_cluster.cc:1428] Setting key 30900135e7422101555d600a3bdbcefa
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:05.827289 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:05.818920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42153 (local address 127.0.98.62:35539)
0928 23:12:05.818980 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:05.818982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:05.819185 (+ 203us) server_negotiation.cc:407] Connection header received
0928 23:12:05.819665 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:05.819666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:05.819675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:05.819729 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:05.820663 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.821197 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.822192 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.822478 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.822546 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:05.824594 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:05.824595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:05.824595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:05.824604 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:05.826030 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.826406 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.826407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.826408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.826470 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.826688 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.826689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.826689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.826822 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:05.826892 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:05.827111 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 23:12:05.827194 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:12:05.827358 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:05.818896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:42153)
0928 23:12:05.819063 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:05.819105 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:05.819584 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:05.819787 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:05.819791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:05.820345 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:05.820562 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:05.820567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.821297 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.821298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:05.822113 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:05.822115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.822554 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.822555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:05.822684 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:05.823185 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:05.823194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:05.824502 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:05.826100 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:05.826106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:05.826108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:05.826313 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:05.826535 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:05.826535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:05.826535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:05.826637 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:05.826878 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:05.826881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:05.826995 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:12:05.827070 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 23:12:05.827192 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 23:12:05.827817 6670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:05.827901 6670 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:05.828065 6670 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:12:05.828573 6479 ts_manager.cc:194] Registered new tserver with Master: a8d060f45c3d4d3bb655319ccf49d21f (127.0.98.1:41953)
I20260928 23:12:05.829170 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 23:12:05.844110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:05.922036 6675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:05.922250 6675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:05.922281 6675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:05.923440 6675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:05.923481 6675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:05.923516 6675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:05.924918 6675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:05.925153 6675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:05.925377 6675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:05.929924 6683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:05.929979 6675 server_base.cc:1034] running on GCE node
W20260928 23:12:05.930044 6680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:05.929929 6681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:05.930318 6675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:05.930513 6675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:05.931648 6675 hybrid_clock.cc:648] HybridClock initialized: now 1790637125931638 us; error 33 us; skew 500 ppm
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:05.933904 6675 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:12:05.934290 6675 webserver.cc:460] Webserver started at http://127.0.98.2:38661/ using document root <none> and password file <none>
I20260928 23:12:05.934456 6675 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:05.934494 6675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:05.934585 6675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:05.935551 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c35975b75b54ff7a0591a43205d3a54"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1c719a22408d24760d2a9cd675e7a1dc"
server_key_iv: "862c7ae8cae66787ebacfb36e98524fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.935850 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c35975b75b54ff7a0591a43205d3a54"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1c719a22408d24760d2a9cd675e7a1dc"
server_key_iv: "862c7ae8cae66787ebacfb36e98524fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.936960 6675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:12:05.937604 6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.937747 6675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.937813 6675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "2c35975b75b54ff7a0591a43205d3a54"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1c719a22408d24760d2a9cd675e7a1dc"
server_key_iv: "862c7ae8cae66787ebacfb36e98524fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:05.937882 6675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:05.951417 6675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:05.953114 6675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:05.953269 6675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:05.953511 6675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:05.953825 6675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:05.953858 6675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.953879 6675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:05.953902 6675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:05.959250 6675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38425
I20260928 23:12:05.959312 6803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38425 every 8 connection(s)
I20260928 23:12:05.959584 6675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:12:05 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:05.967808 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:05.960227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54485 (local address 127.0.98.62:35539)
0928 23:12:05.960283 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:05.960285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:05.960481 (+ 196us) server_negotiation.cc:407] Connection header received
0928 23:12:05.960985 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:05.960987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:05.960998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:05.961052 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:05.962031 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.962523 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.963483 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.963668 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.963724 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:05.965774 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:05.965776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:05.965776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:05.965783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:05.966797 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.967124 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.967125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.967125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.967167 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:05.967334 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:05.967334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:05.967335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:05.967426 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:05.967469 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:05.967678 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 23:12:05.967735 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:12:05.968003 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:05.960211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:54485)
0928 23:12:05.960363 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:05.960399 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:05.960889 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:05.961125 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:05.961129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:05.961737 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:05.961934 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:05.961939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.962610 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.962611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:05.963400 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:05.963402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:05.963735 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:05.963736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:05.963862 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:05.964309 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:05.964318 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:05.965686 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:05.966859 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:05.966864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:05.966866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:05.967071 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:05.967208 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:05.967209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:05.967209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:05.967295 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:05.967465 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:05.967467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:05.967580 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 23:12:05.967662 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:12:05.967779 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260928 23:12:05.968161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6675
I20260928 23:12:05.968257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:05.968420 6804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:05.968456 392 external_mini_cluster.cc:1428] Setting key 365bb0086aa70e5c2700b6fc5fcd8bf6
I20260928 23:12:05.968504 6804 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:05.968701 6804 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260928 23:12:05.969041 6479 ts_manager.cc:194] Registered new tserver with Master: 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
I20260928 23:12:05.969441 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54485
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:12:05.992096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:06.069224 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.069414 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.069435 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.070492 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.070528 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.070552 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:06.071888 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:06.072055 6809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.072230 6809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.076800 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.076793 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.076792 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.076884 6809 server_base.cc:1034] running on GCE node
I20260928 23:12:06.077118 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.077356 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.078485 6809 hybrid_clock.cc:648] HybridClock initialized: now 1790637126078476 us; error 25 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.080672 6809 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:12:06.081080 6809 webserver.cc:460] Webserver started at http://127.0.98.3:42797/ using document root <none> and password file <none>
I20260928 23:12:06.081305 6809 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.081352 6809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.081476 6809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:06.082455 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b1d17ac801c4df7b815e158e52a3c17"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "d6940da070a75ded2d51a5da9eae46c0"
server_key_iv: "61c5d91d0720c47ef7a7899564be4cd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.082782 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b1d17ac801c4df7b815e158e52a3c17"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "d6940da070a75ded2d51a5da9eae46c0"
server_key_iv: "61c5d91d0720c47ef7a7899564be4cd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.083922 6809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:12:06.084515 6824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.084657 6809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.084733 6809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "7b1d17ac801c4df7b815e158e52a3c17"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "d6940da070a75ded2d51a5da9eae46c0"
server_key_iv: "61c5d91d0720c47ef7a7899564be4cd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.084798 6809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.103971 6809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.105679 6809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:06.105810 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.106015 6809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:06.106297 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:06.106333 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.106361 6809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:06.106377 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.111627 6809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41905
I20260928 23:12:06.111670 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41905 every 8 connection(s)
I20260928 23:12:06.111954 6809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:06.116575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6809
I20260928 23:12:06.116670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:06.116861 392 external_mini_cluster.cc:1428] Setting key fcbe278a5a8d77c7077b8ff0b4846cea
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:06.120664 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.112615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38753 (local address 127.0.98.62:35539)
0928 23:12:06.112690 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.112692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.112855 (+ 163us) server_negotiation.cc:407] Connection header received
0928 23:12:06.113382 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.113384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.113395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.113446 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.114408 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.114978 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.116032 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.116218 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.116272 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.118439 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.118441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.118441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.118447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.119688 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.119990 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.119990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.119991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.120035 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.120204 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.120204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.120205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.120297 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.120339 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.120536 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.120589 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 23:12:06.120857 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.112575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:38753)
0928 23:12:06.112732 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.112776 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.113285 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.113529 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.113533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.114103 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.114323 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.114328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.115049 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.115050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.115963 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.115965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.116282 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.116283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.116407 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.116982 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.116992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.118363 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.119746 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.119749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.119751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.119939 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.120077 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.120077 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.120078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.120164 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.120337 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.120340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.120449 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.120523 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.120635 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 23:12:06.121277 6938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:06.121361 6938 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.121496 6938 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260928 23:12:06.121807 6479 ts_manager.cc:194] Registered new tserver with Master: 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905)
I20260928 23:12:06.122123 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38753
I20260928 23:12:06.128372 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 23:12:06.149406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46337
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539,127.0.98.61:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:06.227764 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.227967 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.227995 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.229126 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:06.229168 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.229182 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.229204 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:06.229259 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:06.230684 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539,127.0.98.61:46337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
W20260928 23:12:06.230788 6943 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35539,127.0.98.61:46337'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 23:12:06.230926 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.231128 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.235536 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.235594 6943 server_base.cc:1034] running on GCE node
W20260928 23:12:06.235529 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.235536 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.235953 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.236215 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.237342 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790637126237330 us; error 28 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.239524 6943 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 23:12:06.239907 6943 webserver.cc:460] Webserver started at http://127.0.98.61:43005/ using document root <none> and password file <none>
I20260928 23:12:06.240088 6943 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.240134 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.240262 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:06.241324 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/instance:
uuid: "287a0334a837431a8a526aa61539ac05"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "61325ba4309bc4c92d1b9adf3e899c06"
server_key_iv: "f560d8dbc7cb41356e150244165225b0"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.241667 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal/instance:
uuid: "287a0334a837431a8a526aa61539ac05"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "61325ba4309bc4c92d1b9adf3e899c06"
server_key_iv: "f560d8dbc7cb41356e150244165225b0"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.242818 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:06.243430 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.243574 6943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:06.243649 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "287a0334a837431a8a526aa61539ac05"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "61325ba4309bc4c92d1b9adf3e899c06"
server_key_iv: "f560d8dbc7cb41356e150244165225b0"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.243711 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.271301 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.273243 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:06.273396 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.276918 6943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46337
I20260928 23:12:06.276963 7010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46337 every 8 connection(s)
I20260928 23:12:06.277297 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:12:06.278240 7011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:06.279661 7011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:12:06.283489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 6943
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:06.288111 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.280321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.98.62:35539)
0928 23:12:06.280368 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.280370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.280538 (+ 168us) server_negotiation.cc:407] Connection header received
0928 23:12:06.280967 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.280968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.280977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.281028 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.281608 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.282078 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.282972 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.283138 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.283187 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.285449 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.285451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.285451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.285458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.286737 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.287079 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.287081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.287081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.287142 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.287385 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.287386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.287386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.287579 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.287630 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.287943 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.288024 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:12:06.288398 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.280280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55490)
0928 23:12:06.280421 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.280461 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.280897 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.281081 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.281084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.281357 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.281541 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.281546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.282138 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.282139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.282917 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.282919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.283194 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.283194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.283286 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.283781 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.283788 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.285374 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.286807 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.286812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.286814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.287010 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.287197 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.287197 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.287198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.287329 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.287638 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.287644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.287809 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.287928 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.288144 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260928 23:12:06.289002 7011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.291904 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.283940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.0.98.61:46337)
0928 23:12:06.284069 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.284074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.284094 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:12:06.284155 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.284157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.284172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.284229 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.284826 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.285718 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.286587 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.286995 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.287048 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.289599 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.289604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.289605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.289609 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.290856 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.291157 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.291160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.291160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.291219 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.291400 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.291401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.291401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.291553 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.291693 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.291775 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.291840 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.295331 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.289236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47434)
0928 23:12:06.289268 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.289279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.289318 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.289507 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.289508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.289629 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.289672 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.289674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.290425 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.290425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.290875 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.290876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.291219 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.291220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.291281 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.291636 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.291640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.292884 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.294637 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.294638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.294638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.294778 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.294925 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.294925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.294925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.294966 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.295113 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.295114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.295144 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.295198 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.295259 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 23:12:06.295418 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.289280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.98.61:46337)
0928 23:12:06.289404 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.289405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.289412 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:06.289430 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.289430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.289437 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.289488 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.289749 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.290354 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.290949 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.291156 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.291216 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.292954 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.292955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.292955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.292960 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.294578 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.294836 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.294837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.294837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.294882 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.294998 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.294998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.294998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.295076 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.295116 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.295215 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.295299 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 23:12:06.296483 7011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Bootstrap starting.
I20260928 23:12:06.297031 7011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:06.297320 7011 log.cc:824] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:06.297943 7011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: No bootstrap required, opened a new log
I20260928 23:12:06.299197 7011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } }
I20260928 23:12:06.299307 7011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:06.299328 7011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287a0334a837431a8a526aa61539ac05, State: Initialized, Role: FOLLOWER
I20260928 23:12:06.299409 7011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } }
I20260928 23:12:06.299640 7011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:06.299602 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } } }
I20260928 23:12:06.299695 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:12:06.301007 7029 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:12:06.301100 7029 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:12:06.302414 6943 master_runner.cc:428] Detected that this master 287a0334a837431a8a526aa61539ac05 is joining an existing cluster
I20260928 23:12:06.302451 6943 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46337
I20260928 23:12:06.308094 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.303246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.0.98.62:35539)
0928 23:12:06.303290 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.303292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.303413 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:12:06.303470 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.303471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.303481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.303526 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.303786 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.304254 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.304924 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.305126 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.305177 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.305975 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.305976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.305976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.305983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.307151 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.307424 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.307425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.307425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.307475 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.307654 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.307655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.307655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.307760 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.307811 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.307939 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.308002 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:12:06.308131 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.303219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55500)
0928 23:12:06.303357 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.303366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.303416 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.303530 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.303530 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.303671 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.303722 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.303723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.304318 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.304319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.304857 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.304858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.305202 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.305203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.305278 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.305646 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.305649 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.305900 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.307221 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.307222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.307222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.307367 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.307552 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.307553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.307553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.307605 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.307809 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.307810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.307849 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.307908 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.308001 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 23:12:06.308473 7018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:06.308559 7018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } }
I20260928 23:12:06.308756 7018 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:06.308938 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "287a0334a837431a8a526aa61539ac05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" is_pre_election: true
I20260928 23:12:06.309276 6962 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:06.309414 7018 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.318524 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.310472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47436 (local address 127.0.98.61:46337)
0928 23:12:06.310509 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.310510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.310685 (+ 175us) server_negotiation.cc:407] Connection header received
0928 23:12:06.310737 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.310738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.310748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.310811 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.311158 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.311890 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.312632 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.312961 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.313024 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.315616 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.315616 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.315617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.315622 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.317291 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.317731 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.317732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.317733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.317786 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.317969 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.317970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.317970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.318099 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.318154 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.318354 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.318429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:12:06.318576 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.310420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47436)
0928 23:12:06.310587 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.310624 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.310683 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.310815 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.310817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.311047 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.311090 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.311092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.311969 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.311970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.312564 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.312566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.313038 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.313038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.313096 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.313573 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.313577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.315239 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.317364 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.317372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.317375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.317632 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.317853 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.317854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.317854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.317911 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.318160 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.318168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.318222 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.318327 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.318452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260928 23:12:06.318722 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.313375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47448)
0928 23:12:06.313438 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.313448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.313486 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.313759 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.313759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.313910 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.313962 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.313964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.314668 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.314669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.315245 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.315247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.315603 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.315604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.315657 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.316034 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.316037 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.316389 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.317839 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.317840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.317840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.318002 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.318181 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.318181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.318182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.318228 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.318469 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.318470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.318500 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.318555 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.318635 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:12:06.319262 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.313579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.0.98.61:46337)
0928 23:12:06.313634 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.313636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.313644 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:06.313672 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.313672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.313687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.313748 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.314042 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.314582 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.315324 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.315522 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.315571 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.316470 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.316472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.316472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.316478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.317772 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.318069 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.318070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.318070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.318117 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.318281 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.318282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.318283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.318394 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.318448 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.319112 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.319186 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:12:06.319315 6479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46337: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "287a0334a837431a8a526aa61539ac05"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46337
}
attrs {
promote: true
}
}
dest_uuid: "81cb14f6bcb1458a834b9e4f35a277d3"
cas_config_opid_index: -1
I20260928 23:12:06.319626 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } }
I20260928 23:12:06.320179 7037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 287a0334a837431a8a526aa61539ac05 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } }
I20260928 23:12:06.320327 6985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:06.321003 7037 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46337
I20260928 23:12:06.321079 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } }
I20260928 23:12:06.321130 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:06.321203 6985 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Refusing update from remote peer 81cb14f6bcb1458a834b9e4f35a277d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 23:12:06.321255 6943 master.cc:537] Master@127.0.98.61:46337 shutting down...
I20260928 23:12:06.321520 7037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 23:12:06.322234 6464 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 23:12:06.322935 6464 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 -> Peer 287a0334a837431a8a526aa61539ac05 (127.0.98.61:46337): Couldn't send request to peer 287a0334a837431a8a526aa61539ac05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:12:06.323850 6943 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:06.323925 6943 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:06.323971 6943 tablet_replica.cc:330] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: stopping tablet replica
I20260928 23:12:06.336438 6943 master.cc:552] Master@127.0.98.61:46337 shutdown complete.
I20260928 23:12:06.336503 6943 master_runner.cc:305] Clearing existing system tablet
I20260928 23:12:06.336629 6943 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.336687 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.337427 6943 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260928 23:12:06.338124 7042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.338263 6943 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.338322 6943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:12:06.338346 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "287a0334a837431a8a526aa61539ac05"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "61325ba4309bc4c92d1b9adf3e899c06"
server_key_iv: "f560d8dbc7cb41356e150244165225b0"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.339143 6943 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:06.340637 6943 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:12:06.340711 6943 log.cc:1196] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 23:12:06.340894 6943 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Deleting consensus metadata
I20260928 23:12:06.341002 6943 master_runner.cc:315] Copying system tablet from 127.0.98.62:35539
I20260928 23:12:06.341603 6943 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35539
I20260928 23:12:06.347453 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.341975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.0.98.62:35539)
0928 23:12:06.342026 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.342028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.342063 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:12:06.342117 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.342118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.342128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.342207 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.342471 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.343251 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.343997 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.344299 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.344370 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.345308 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.345310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.345311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.345319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.346477 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.346777 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.346778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.346779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.346833 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.346991 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.346992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.346993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.347106 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.347160 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.347290 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.347361 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:12:06.347470 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.341896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55502)
0928 23:12:06.342003 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.342013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.342074 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.342211 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.342211 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.342360 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.342413 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.342414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.343346 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.343347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.343923 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.343925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.344374 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.344375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.344426 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.344922 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.344925 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.345237 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.346554 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.346555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.346556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.346715 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.346884 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.346885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.346885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.346943 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.347158 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.347159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.347202 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.347261 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.347361 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":67,"threads_started":1}
I20260928 23:12:06.347893 6499 tablet_copy_service.cc:140] P 81cb14f6bcb1458a834b9e4f35a277d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 287a0334a837431a8a526aa61539ac05 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55502)
I20260928 23:12:06.347955 6499 tablet_copy_service.cc:161] P 81cb14f6bcb1458a834b9e4f35a277d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 287a0334a837431a8a526aa61539ac05 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55502: session id = 287a0334a837431a8a526aa61539ac05-00000000000000000000000000000000
I20260928 23:12:06.348542 6499 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:12:06.349133 6943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:12:06.350365 6943 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: tablet copy: Starting download of 0 data blocks...
I20260928 23:12:06.350512 6943 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: tablet copy: Starting download of 1 WAL segments...
I20260928 23:12:06.351337 6943 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:12:06.352205 6499 tablet_copy_service.cc:342] P 81cb14f6bcb1458a834b9e4f35a277d3: Request end of tablet copy session 287a0334a837431a8a526aa61539ac05-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55502
I20260928 23:12:06.352263 6499 tablet_copy_service.cc:433] P 81cb14f6bcb1458a834b9e4f35a277d3: ending tablet copy session 287a0334a837431a8a526aa61539ac05-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 287a0334a837431a8a526aa61539ac05
I20260928 23:12:06.354592 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.355772 7049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.355782 7050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.355789 7052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.356074 6943 server_base.cc:1034] running on GCE node
I20260928 23:12:06.356172 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.356294 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.357422 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790637126357416 us; error 23 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.358762 6943 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 23:12:06.359014 6943 webserver.cc:460] Webserver started at http://127.0.98.61:41397/ using document root <none> and password file <none>
I20260928 23:12:06.359120 6943 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.359153 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.359786 6943 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:06.360294 7059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.360399 6943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.360432 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "287a0334a837431a8a526aa61539ac05"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "61325ba4309bc4c92d1b9adf3e899c06"
server_key_iv: "f560d8dbc7cb41356e150244165225b0"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.360487 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.369568 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.371037 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.373385 6943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46337
I20260928 23:12:06.373739 7111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46337 every 8 connection(s)
I20260928 23:12:06.373819 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:12:06.374657 7112 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:12:06.375188 7112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Bootstrap starting.
I20260928 23:12:06.378003 7112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:12:06.378235 7112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: Bootstrap complete.
I20260928 23:12:06.378432 7112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } }
I20260928 23:12:06.378523 7112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 287a0334a837431a8a526aa61539ac05, State: Initialized, Role: LEARNER
I20260928 23:12:06.378568 7112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } }
I20260928 23:12:06.378760 7112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:06.378743 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } }
I20260928 23:12:06.378832 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:12:06.380223 7126 catalog_manager.cc:1259] Loaded cluster ID: a321c9f99cfe4dcab04c616119533bb2
I20260928 23:12:06.380270 7126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: loading cluster ID for follower catalog manager: success
I20260928 23:12:06.381129 7126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: acquiring CA information for follower catalog manager: success
I20260928 23:12:06.381422 7126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:12:06.399966 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.393278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.98.61:46337)
0928 23:12:06.393403 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.393406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.393416 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:12:06.393444 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.393444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.393457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.393562 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.394052 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.395117 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.395783 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.396113 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.396182 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.397318 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.397319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.397320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.397327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.398827 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.399122 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.399123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.399123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.399189 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.399329 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.399329 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.399330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.399456 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.399548 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.399628 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.399705 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 23:12:06.424048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35359
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:06.501829 7130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.502009 7130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.502029 7130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.503110 7130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:06.503147 7130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.503165 7130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.503187 7130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:06.503209 7130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:06.504604 7130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:06.504827 7130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.505018 7130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.509377 7136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.509343 7135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.509343 7138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.509989 7130 server_base.cc:1034] running on GCE node
I20260928 23:12:06.510210 7130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.510468 7130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.511617 7130 hybrid_clock.cc:648] HybridClock initialized: now 1790637126511598 us; error 26 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.513993 7130 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 23:12:06.514424 7130 webserver.cc:460] Webserver started at http://127.0.98.60:38899/ using document root <none> and password file <none>
I20260928 23:12:06.514616 7130 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.514675 7130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.514798 7130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:06.515908 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/instance:
uuid: "a3b2c868b32145d68bd0d861457885d6"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "3a65f664361dfa5d146648538135d1d8"
server_key_iv: "f2b2b6ace3ee66bf0ca15d9670b51e59"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.516230 7130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal/instance:
uuid: "a3b2c868b32145d68bd0d861457885d6"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "3a65f664361dfa5d146648538135d1d8"
server_key_iv: "f2b2b6ace3ee66bf0ca15d9670b51e59"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.517411 7130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:06.518074 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.518217 7130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.518292 7130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "a3b2c868b32145d68bd0d861457885d6"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "3a65f664361dfa5d146648538135d1d8"
server_key_iv: "f2b2b6ace3ee66bf0ca15d9670b51e59"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.518356 7130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.541345 7130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.543066 7130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:06.543198 7130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.547227 7130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35359
I20260928 23:12:06.547261 7197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35359 every 8 connection(s)
I20260928 23:12:06.547580 7130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:12:06.548280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 7130
I20260928 23:12:06.548501 7198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:06.550738 7198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:06.557946 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.548825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.0.98.60:35359)
0928 23:12:06.548970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.548980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.549005 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:12:06.549564 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.549571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.549627 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.549741 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.550443 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.551461 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.552124 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.552534 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.552623 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.554967 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.554974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.554975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.554982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.556549 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.556842 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.556845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.556845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.556936 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.557109 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.557109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.557110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.557393 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.557506 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.557609 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.557736 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260928 23:12:06.558055 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.551288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55518)
0928 23:12:06.551457 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.551498 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.551565 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.551713 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.551716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.551950 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.552010 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.552012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.552644 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.552645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.553355 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.553357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.553678 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.553679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.553758 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.554143 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.554152 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.555634 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.557007 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.557013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.557015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.557233 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.557401 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.557401 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.557402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.557458 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.557746 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.557750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.557804 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.557884 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.557973 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260928 23:12:06.558073 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.551427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.0.98.62:35539)
0928 23:12:06.551470 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.551472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.551551 (+ 79us) server_negotiation.cc:407] Connection header received
0928 23:12:06.551622 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.551623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.551632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.551680 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.552086 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.552559 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.553418 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.553614 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.553667 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.555717 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.555718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.555719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.555726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.556942 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.557298 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.557300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.557300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.557351 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.557524 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.557525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.557525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.557657 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.557720 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.557922 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.557981 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:12:06.558475 7198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.563880 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.558707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47492)
0928 23:12:06.558749 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.558759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.558799 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.558967 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.558968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.559073 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.559112 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.559113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.559703 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.559704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.560223 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.560224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.560550 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.560551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.560611 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.560884 (+ 273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.560887 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.561916 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.563057 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.563058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.563059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.563201 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.563367 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.563367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.563368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.563418 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.563616 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.563616 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.563653 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.563712 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.563792 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:12:06.563901 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.558823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.0.98.61:46337)
0928 23:12:06.558871 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.558873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.558880 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:06.558901 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.558902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.558913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.558958 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.559178 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.559641 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.560288 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.560499 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.560549 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.561996 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.561997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.561997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.562004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.562989 (+ 985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.563266 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.563268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.563268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.563317 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.563465 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.563466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.563466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.563565 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.563617 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.563745 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.563818 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:12:06.564663 7198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35359 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 23:12:06.569515 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.564861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.0.1:45728)
0928 23:12:06.564929 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.564936 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.564966 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.565049 (+ 83us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.565050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.565140 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.565163 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.565164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.565733 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.565733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.566175 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.566176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.566450 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.566451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.566484 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.566736 (+ 252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.566739 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.567686 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.568775 (+ 1089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.568775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.568776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.568911 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.569075 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.569075 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.569075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.569112 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.569308 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.569309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.569339 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.569382 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.569445 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 23:12:06.569567 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.564900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.0.98.60:35359)
0928 23:12:06.564948 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.564950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.564958 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:06.565007 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.565008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.565014 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.565046 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.565249 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.565679 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.566239 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.566408 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.566448 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.567751 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.567752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.567752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.567757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.568717 (+ 960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.568976 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.568977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.568977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.569026 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.569143 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.569143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.569143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.569270 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.569314 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.569404 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.569486 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":32}
I20260928 23:12:06.570758 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Bootstrap starting.
I20260928 23:12:06.571520 7198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:06.571753 7198 log.cc:824] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:06.572353 7198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: No bootstrap required, opened a new log
I20260928 23:12:06.573745 7198 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35359 } }
I20260928 23:12:06.573873 7198 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:06.573904 7198 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b2c868b32145d68bd0d861457885d6, State: Initialized, Role: FOLLOWER
I20260928 23:12:06.574018 7198 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35359 } }
I20260928 23:12:06.574245 7203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35359 } } }
I20260928 23:12:06.574321 7198 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:06.574339 7203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:12:06.576262 7214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:12:06.576471 7214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:12:06.578558 7130 master_runner.cc:428] Detected that this master a3b2c868b32145d68bd0d861457885d6 is joining an existing cluster
I20260928 23:12:06.578599 7130 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35359
I20260928 23:12:06.604869 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.579949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.0.98.61:46337)
0928 23:12:06.580049 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.580051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.580059 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:06.580079 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.580080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.580090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.580163 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.580754 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.581488 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582489 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582789 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582825 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.602341 (+ 19516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.602343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.602343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.602351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.603714 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604041 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.604043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.604043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.604103 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604300 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.604301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.604301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.604441 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.604481 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.604685 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.604774 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 23:12:06.604946 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.580227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.0.98.60:35359)
0928 23:12:06.580265 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.580266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.580274 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:06.580292 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.580292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.580300 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.580349 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.580675 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.581438 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582496 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582778 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582815 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.583889 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.583890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.583890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.583896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.603140 (+ 19244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604277 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.604278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.604278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.604320 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604488 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.604488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.604489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.604590 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.604624 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.604774 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.604836 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:12:06.605080 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.579838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.0.1:45734)
0928 23:12:06.579933 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.579942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.579974 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.580364 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.580365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.580534 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.580612 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.580614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.581528 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.581529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.582405 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.582408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582892 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.582948 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.583380 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.583386 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.583797 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.603797 (+ 20000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.603798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.603798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.604201 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.604377 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.604378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.604432 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.604659 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.604659 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.604690 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.604900 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.604981 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":24,"thread_start_us":64,"threads_started":1}
I20260928 23:12:06.605749 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.579580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47504)
0928 23:12:06.579735 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.579747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.579801 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.580283 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.580284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.580614 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.580679 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.580681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.581562 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.581562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.582407 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.582409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582895 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.582953 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.601723 (+ 18770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.601729 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.602233 (+ 504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.603785 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.603786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.603786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.603983 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.604169 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.604170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.604225 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.604523 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.604523 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.604565 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.605539 (+ 974us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.605643 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":35,"thread_start_us":99,"threads_started":1}
I20260928 23:12:06.606118 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.580556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.0.98.62:35539)
0928 23:12:06.580603 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.580605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.581597 (+ 992us) server_negotiation.cc:407] Connection header received
0928 23:12:06.581683 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.581684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.581694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.581722 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.582058 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582757 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.583525 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.583730 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.583782 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.602855 (+ 19073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.602857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.602858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.602866 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.604793 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.605365 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.605366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.605367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.605422 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.605601 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.605602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.605602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.605737 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.605786 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.605959 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.606032 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:12:06.606470 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.580781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55532)
0928 23:12:06.581548 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.581556 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.581620 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.581774 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.581775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.581933 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.581980 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.581981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.582839 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.582840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.583453 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.583455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.583787 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.583788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.583843 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.584209 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.584213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.602745 (+ 18532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.604880 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.604881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.604882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.605280 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.605488 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.605489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.605489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.605540 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.605816 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.605817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.605859 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.606270 (+ 411us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.606370 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":68,"threads_started":1}
I20260928 23:12:06.609418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6541
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 23:12:06.614480 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.608346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.0.1:45740)
0928 23:12:06.608527 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.608536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.608580 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.608705 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.608706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.608837 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.608873 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.608874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.609952 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.609953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.610499 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.610502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.610916 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.610916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.610958 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.611265 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.611269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.612300 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.613705 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.613706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.613707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.613851 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.614017 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.614017 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.614017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.614058 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.614261 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.614262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.614304 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.614353 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.614420 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 23:12:06.614521 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.608402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.0.98.60:35359)
0928 23:12:06.608442 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.608443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.608579 (+ 136us) server_negotiation.cc:407] Connection header received
0928 23:12:06.608643 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.608644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.608653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.608703 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.608930 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.609547 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.610580 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.610859 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.610921 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.612374 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.612376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.612376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.612382 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.613637 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.613911 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.613911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.613912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.613963 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.614103 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.614103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.614103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.614207 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.614275 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.614375 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.614443 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:12:06.614923 6479 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35359: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a3b2c868b32145d68bd0d861457885d6"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35359
}
attrs {
promote: true
}
}
dest_uuid: "81cb14f6bcb1458a834b9e4f35a277d3"
cas_config_opid_index: 5
I20260928 23:12:06.615152 6479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:06.615293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41953
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35539
--webserver_port=34845
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:12:06.615470 7037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3b2c868b32145d68bd0d861457885d6 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.616328 7037 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35359
I20260928 23:12:06.616403 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.616487 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:06.616871 7172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:06.617813 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 1 FOLLOWER]: Refusing update from remote peer 81cb14f6bcb1458a834b9e4f35a277d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:12:06.618074 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:06.618218 7130 master.cc:537] Master@127.0.98.60:35359 shutting down...
W20260928 23:12:06.618722 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 -> Peer a3b2c868b32145d68bd0d861457885d6 (127.0.98.60:35359): Couldn't send request to peer a3b2c868b32145d68bd0d861457885d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:12:06.620901 7130 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:06.621001 7130 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:06.621042 7130 tablet_replica.cc:330] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: stopping tablet replica
I20260928 23:12:06.621922 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.616554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.0.1:47506)
0928 23:12:06.617060 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.617071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.617113 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.617361 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.617362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.617363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:06.617398 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.617399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.619462 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.619464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.620676 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.620679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.621661 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.621662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.621739 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.621761 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.621814 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.621855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488}
I20260928 23:12:06.622431 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.616689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.98.61:46337)
0928 23:12:06.617176 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.617178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.617185 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:06.617229 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.617230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.617253 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.617306 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:06.617457 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.618127 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.620771 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.621390 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.621453 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.621518 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.622209 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.622256 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260928 23:12:06.622949 7086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 LEARNER]: Refusing update from remote peer 81cb14f6bcb1458a834b9e4f35a277d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 23:12:06.623180 7038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 23:12:06.623795 7130 master.cc:552] Master@127.0.98.60:35359 shutdown complete.
I20260928 23:12:06.623701 7086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a3b2c868b32145d68bd0d861457885d6 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.623826 7130 master_runner.cc:305] Clearing existing system tablet
I20260928 23:12:06.623944 7130 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.623978 7130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.624758 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.624837 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:12:06.624953 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81cb14f6bcb1458a834b9e4f35a277d3. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: true } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.625034 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:12:06.625733 7038 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: attempting to promote NON_VOTER 287a0334a837431a8a526aa61539ac05 to VOTER
I20260928 23:12:06.625928 7038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:06.626576 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.626437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.0.1:45756)
0928 23:12:06.626514 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.626550 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 23:12:06.626837 7086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 LEARNER]: Refusing update from remote peer 81cb14f6bcb1458a834b9e4f35a277d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 23:12:06.627271 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.627186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.0.1:45772)
0928 23:12:06.627224 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.627246 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:12:06.627363 7130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
W20260928 23:12:06.627676 6465 consensus_peers.cc:487] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 -> Peer a3b2c868b32145d68bd0d861457885d6 (127.0.98.60:35359): Couldn't send request to peer a3b2c868b32145d68bd0d861457885d6. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 23:12:06.627753 7227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 23:12:06.627903 7228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.627990 7130 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.628023 7130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.628043 7130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "a3b2c868b32145d68bd0d861457885d6"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "3a65f664361dfa5d146648538135d1d8"
server_key_iv: "f2b2b6ace3ee66bf0ca15d9670b51e59"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.628449 7037 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 287a0334a837431a8a526aa61539ac05 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.628777 7130 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:06.629340 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.629410 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:06.629733 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.629801 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:06.630014 7130 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 23:12:06.630074 7130 log.cc:1196] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 23:12:06.630060 7086 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 287a0334a837431a8a526aa61539ac05 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.630244 7130 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Deleting consensus metadata
I20260928 23:12:06.630481 7130 master_runner.cc:315] Copying system tablet from 127.0.98.62:35539
I20260928 23:12:06.630868 7115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.630939 7115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:12:06.631029 7130 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35539
I20260928 23:12:06.644003 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.633444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55544)
0928 23:12:06.637299 (+ 3855us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.637311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.637375 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.637516 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.637517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.637722 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.638066 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.638069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.638994 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.638996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.639827 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.639830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.640293 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.640294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.640359 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.640834 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.640838 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.641200 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.642989 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.642990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.642991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.643197 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.643393 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.643393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.643394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.643454 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.643698 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.643699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.643743 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.643808 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.643905 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3833,"thread_start_us":69,"threads_started":1}
I20260928 23:12:06.644004 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.633491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.0.98.62:35539)
0928 23:12:06.633529 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.633531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.637377 (+ 3846us) server_negotiation.cc:407] Connection header received
0928 23:12:06.637432 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.637433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.637443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.637504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.638145 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.638916 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.639901 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.640225 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.640294 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.641452 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.641453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.641454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.641460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.642915 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.643267 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.643268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.643269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.643330 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.643509 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.643509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.643510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.643637 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.643700 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.643841 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.643916 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:06.644693 6499 tablet_copy_service.cc:140] P 81cb14f6bcb1458a834b9e4f35a277d3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a3b2c868b32145d68bd0d861457885d6 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55544)
I20260928 23:12:06.644788 6499 tablet_copy_service.cc:161] P 81cb14f6bcb1458a834b9e4f35a277d3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a3b2c868b32145d68bd0d861457885d6 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55544: session id = a3b2c868b32145d68bd0d861457885d6-00000000000000000000000000000000
I20260928 23:12:06.645383 6499 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:12:06.645843 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 23:12:06.647123 7130 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: tablet copy: Starting download of 0 data blocks...
I20260928 23:12:06.647259 7130 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: tablet copy: Starting download of 1 WAL segments...
I20260928 23:12:06.648046 7130 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:12:06.648926 6499 tablet_copy_service.cc:342] P 81cb14f6bcb1458a834b9e4f35a277d3: Request end of tablet copy session a3b2c868b32145d68bd0d861457885d6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:55544
I20260928 23:12:06.648988 6499 tablet_copy_service.cc:433] P 81cb14f6bcb1458a834b9e4f35a277d3: ending tablet copy session a3b2c868b32145d68bd0d861457885d6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a3b2c868b32145d68bd0d861457885d6
I20260928 23:12:06.653910 7130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.655028 7239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.655059 7236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.655651 7130 server_base.cc:1034] running on GCE node
W20260928 23:12:06.656002 7237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.656185 7130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.656334 7130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.657465 7130 hybrid_clock.cc:648] HybridClock initialized: now 1790637126657453 us; error 29 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.658912 7130 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 23:12:06.659152 7130 webserver.cc:460] Webserver started at http://127.0.98.60:44299/ using document root <none> and password file <none>
I20260928 23:12:06.659260 7130 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.659291 7130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.660000 7130 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:06.660568 7246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.660677 7130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.660725 7130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "a3b2c868b32145d68bd0d861457885d6"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "3a65f664361dfa5d146648538135d1d8"
server_key_iv: "f2b2b6ace3ee66bf0ca15d9670b51e59"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.660782 7130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.670001 7130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.671742 7130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.674715 7130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35359
I20260928 23:12:06.675098 7130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:12:06.677323 7298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35359 every 8 connection(s)
I20260928 23:12:06.678061 7299 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:12:06.678579 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Bootstrap starting.
I20260928 23:12:06.686869 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:12:06.687073 7299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: Bootstrap complete.
I20260928 23:12:06.687328 7299 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:06.687417 7299 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3b2c868b32145d68bd0d861457885d6, State: Initialized, Role: LEARNER
I20260928 23:12:06.687451 7299 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:06.687616 7299 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:06.687634 7302 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } } }
I20260928 23:12:06.687830 7302 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6 [sys.catalog]: This master's current role is: LEARNER
I20260928 23:12:06.689270 7313 catalog_manager.cc:1259] Loaded cluster ID: a321c9f99cfe4dcab04c616119533bb2
I20260928 23:12:06.689915 7313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: loading cluster ID for follower catalog manager: success
I20260928 23:12:06.691049 7313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: acquiring CA information for follower catalog manager: success
I20260928 23:12:06.691342 7313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a3b2c868b32145d68bd0d861457885d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 23:12:06.710558 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.710767 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.710798 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.711920 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.711966 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.712001 7223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:06.713435 7223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34845
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:06.713673 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.713869 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.718938 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.718936 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.719125 7223 server_base.cc:1034] running on GCE node
W20260928 23:12:06.719281 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.719460 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.719727 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.720870 7223 hybrid_clock.cc:648] HybridClock initialized: now 1790637126720859 us; error 31 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.723260 7223 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:12:06.723732 7223 webserver.cc:460] Webserver started at http://127.0.98.1:34845/ using document root <none> and password file <none>
I20260928 23:12:06.723969 7223 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.724035 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.725595 7223 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:06.726183 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.726336 7223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:06.726406 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "a8d060f45c3d4d3bb655319ccf49d21f"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1aba2b1fcd680b2b7f774a2011f1e4d0"
server_key_iv: "efedf0d8c8be6d245ce8f2d29254eeb9"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.726733 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.737998 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.739617 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:06.739722 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.739890 7223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:06.740129 7223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:06.740157 7223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.740177 7223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:06.740192 7223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.745855 7223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41953
I20260928 23:12:06.745922 7441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41953 every 8 connection(s)
I20260928 23:12:06.746223 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:06.751428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 7223
I20260928 23:12:06.751583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6675
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.758343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38425
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35539
--webserver_port=38661
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:12:06.760824 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.746992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32841 (local address 127.0.98.60:35359)
0928 23:12:06.747120 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.747122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.747333 (+ 211us) server_negotiation.cc:407] Connection header received
0928 23:12:06.748647 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.748649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.748662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.748718 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.749442 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750441 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.751694 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.752014 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.752048 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.755936 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.755938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.755939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.755947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.758028 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.758765 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.758766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.758767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.758825 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.759308 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.759309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.759310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.759483 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.759534 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.760636 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.760725 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 23:12:06.761390 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.746869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.98.1:32841)
0928 23:12:06.747206 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.747236 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.748563 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.748793 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.748797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.749137 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.749365 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.749370 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750508 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.750509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.751574 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.751575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.752124 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.752125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.752260 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.752886 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.752891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.755794 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.758388 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.758395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.758398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.758680 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.759107 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.759108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.759108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.759240 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.759579 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.759786 (+ 207us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.760215 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.760943 (+ 728us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.761125 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":6,"thread_start_us":94,"threads_started":1}
I20260928 23:12:06.762485 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.747100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53959 (local address 127.0.98.62:35539)
0928 23:12:06.747145 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.747146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.747458 (+ 312us) server_negotiation.cc:407] Connection header received
0928 23:12:06.748677 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.748678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.748690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.748719 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.749479 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750239 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.751691 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.752000 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.752041 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.759675 (+ 7634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.759677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.759677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.759685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.761178 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761655 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.761657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.761657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.761713 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761912 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.761913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.761913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.762083 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.762149 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.762325 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.762405 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:12:06.762717 7442 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35359
I20260928 23:12:06.762815 7442 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.763005 7442 heartbeater.cc:511] Master 127.0.98.60:35359 requested a full tablet report, sending...
I20260928 23:12:06.763003 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.746869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:53959)
0928 23:12:06.747388 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.747396 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.748571 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.748794 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.748798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.749117 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.749367 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.749373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750338 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.750339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.751561 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.751563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.752121 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.752122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.752260 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.752978 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.752982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.758910 (+ 5928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.761312 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.761313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.761530 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.761783 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.761784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.761840 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.762175 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.762176 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.762217 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.762597 (+ 380us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.762695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":195,"thread_start_us":51,"threads_started":1}
I20260928 23:12:06.763667 7263 ts_manager.cc:194] Registered new tserver with Master: a8d060f45c3d4d3bb655319ccf49d21f (127.0.98.1:41953)
I20260928 23:12:06.763813 7443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:06.763862 7443 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.763919 7443 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260928 23:12:06.764222 6479 ts_manager.cc:194] Re-registered known tserver with Master: a8d060f45c3d4d3bb655319ccf49d21f (127.0.98.1:41953)
I20260928 23:12:06.764292 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.746869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.98.1:54539)
0928 23:12:06.747201 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.747236 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.748552 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.748793 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.748798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.749117 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.749367 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.749373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750932 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.750933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.751563 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.751566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.752119 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.752120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.752260 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.752854 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.752864 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.758387 (+ 5523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.760744 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.760745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.760745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.760937 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.761148 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.761149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.761205 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.764013 (+ 2808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.764013 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.764052 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.764110 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.764198 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":11,"thread_start_us":80,"threads_started":1}
I20260928 23:12:06.764344 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.746940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54539 (local address 127.0.98.61:46337)
0928 23:12:06.747108 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.747110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.747315 (+ 205us) server_negotiation.cc:407] Connection header received
0928 23:12:06.748671 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.748672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.748685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.748715 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.750336 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.750862 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.751661 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.751941 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.751979 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.758619 (+ 6640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.758620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.758621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.758629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.760660 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761019 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.761020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.761021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.761077 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.761291 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.761293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.761293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.763921 (+ 2628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.763966 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.764179 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.764257 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 23:12:06.764667 7444 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46337
I20260928 23:12:06.764676 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53959
I20260928 23:12:06.764703 7444 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.764757 7444 heartbeater.cc:511] Master 127.0.98.61:46337 requested a full tablet report, sending...
I20260928 23:12:06.765115 7076 ts_manager.cc:194] Registered new tserver with Master: a8d060f45c3d4d3bb655319ccf49d21f (127.0.98.1:41953)
W20260928 23:12:06.845296 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.845490 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.845512 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.846678 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.846722 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.846748 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:06.848147 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38661
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:06.848325 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.848500 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:06.853135 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:06.853102 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.853168 7450 server_base.cc:1034] running on GCE node
W20260928 23:12:06.853066 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:06.853404 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:06.853597 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:06.854735 7450 hybrid_clock.cc:648] HybridClock initialized: now 1790637126854725 us; error 25 us; skew 500 ppm
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:06.857048 7450 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:12:06.857478 7450 webserver.cc:460] Webserver started at http://127.0.98.2:38661/ using document root <none> and password file <none>
I20260928 23:12:06.857664 7450 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:06.857723 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:06.858929 7450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:06.859408 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.859550 7450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.859625 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "2c35975b75b54ff7a0591a43205d3a54"
format_stamp: "Formatted at 2026-09-28 23:12:05 on dist-test-slave-6vgd"
server_key: "1c719a22408d24760d2a9cd675e7a1dc"
server_key_iv: "862c7ae8cae66787ebacfb36e98524fd"
server_key_version: "encryptionkey@0"
I20260928 23:12:06.859952 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:06.889310 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:06.891544 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:06.891683 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:06.891917 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:06.892287 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:06.892326 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.892362 7450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:06.892391 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:06.897959 7450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38425
I20260928 23:12:06.898056 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38425 every 8 connection(s)
I20260928 23:12:06.898340 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:12:06 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 23:12:06.906716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 7450
I20260928 23:12:06.906870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6809
I20260928 23:12:06.908541 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44723 (local address 127.0.98.62:35539)
0928 23:12:06.899531 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.899533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.899542 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:06.900554 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.900556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.900567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.900624 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.901305 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.901911 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.902885 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903093 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903150 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.905288 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.905289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.905290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.905297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.906760 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.907671 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.907672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.907672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.907728 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.907940 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.907941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.907941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.908053 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.908108 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.908376 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.908454 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:12:06.908697 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:44723)
0928 23:12:06.899192 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.899232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.900224 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.900667 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.900668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.900903 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.901165 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.901172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.902020 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.902021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.902802 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.902804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903213 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.903358 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.903862 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.903872 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.905157 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.906846 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.906853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.906855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.907600 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.907781 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.907782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.907782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.907887 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.908105 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.908110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.908247 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.908345 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.908492 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":8,"thread_start_us":87,"threads_started":1}
I20260928 23:12:06.909157 7580 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:06.909269 7580 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.909425 7580 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260928 23:12:06.909770 6479 ts_manager.cc:194] Re-registered known tserver with Master: 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
I20260928 23:12:06.910084 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44723
I20260928 23:12:06.913199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41905
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35539
--webserver_port=42797
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:12:06.914870 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40811 (local address 127.0.98.60:35359)
0928 23:12:06.899629 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.899631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.899638 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:06.900446 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.900447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.900458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.900505 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.901297 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.902063 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903142 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903423 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903490 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.906033 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.906034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.906034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.906042 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.913726 (+ 7684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914046 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.914047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.914047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.914111 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914294 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.914295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.914295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.914423 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.914484 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.914685 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.914779 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:12:06.914908 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.98.2:40811)
0928 23:12:06.899464 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.899473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.900235 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.900519 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.900520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.900856 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.901165 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.901171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.902150 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.902150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.903074 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.903076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903489 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.903534 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.904083 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.904088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.905937 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.913806 (+ 7869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.913807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.913808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.913983 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.914166 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.914167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.914236 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.914512 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.914513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.914559 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.914654 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.914758 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":65,"thread_start_us":68,"threads_started":1}
I20260928 23:12:06.915287 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37495 (local address 127.0.98.61:46337)
0928 23:12:06.899075 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:06.899077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:06.899320 (+ 243us) server_negotiation.cc:407] Connection header received
0928 23:12:06.900325 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:06.900327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:06.900339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:06.900391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:06.901297 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.902059 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903543 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903777 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903839 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.906032 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:06.906033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:06.906034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:06.906041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:06.914020 (+ 7979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914333 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.914334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.914334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.914392 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914583 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:06.914584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:06.914584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:06.914712 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:06.914753 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:06.915105 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 23:12:06.915200 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:06.915482 7579 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35359
I20260928 23:12:06.915539 7579 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.915621 7579 heartbeater.cc:511] Master 127.0.98.60:35359 requested a full tablet report, sending...
I20260928 23:12:06.915716 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:06.899037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.98.2:37495)
0928 23:12:06.899190 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:06.899232 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:06.900232 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:06.900446 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:06.900450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:06.900856 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:06.901165 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.901171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.902150 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.902150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.903066 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:06.903069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:06.903847 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:06.903848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:06.903908 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:06.904527 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:06.904531 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:06.905962 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:06.914122 (+ 8160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.914123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.914276 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.914462 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:06.914463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:06.914464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:06.914517 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:06.914798 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:06.914798 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:06.914840 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 23:12:06.915453 (+ 613us) client_negotiation.cc:240] Negotiation successful
0928 23:12:06.915568 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":52,"thread_start_us":96,"threads_started":1}
I20260928 23:12:06.915956 7263 ts_manager.cc:194] Registered new tserver with Master: 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
I20260928 23:12:06.916008 7581 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46337
I20260928 23:12:06.916056 7581 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:06.916105 7581 heartbeater.cc:511] Master 127.0.98.61:46337 requested a full tablet report, sending...
I20260928 23:12:06.916352 7076 ts_manager.cc:194] Registered new tserver with Master: 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
W20260928 23:12:06.993976 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:06.994184 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:06.994215 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:06.995321 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:06.995361 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:06.995395 7586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:06.996894 7586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42797
--tserver_master_addrs=127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:06.997144 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:06.997380 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:07.002184 7591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:07.002204 7592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:07.002197 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:07.002398 7586 server_base.cc:1034] running on GCE node
I20260928 23:12:07.002588 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:07.002841 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:07.004020 7586 hybrid_clock.cc:648] HybridClock initialized: now 1790637127004007 us; error 29 us; skew 500 ppm
Sep 28 23:12:07 dist-test-slave-6vgd krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:07.006389 7586 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:12:07.006817 7586 webserver.cc:460] Webserver started at http://127.0.98.3:42797/ using document root <none> and password file <none>
I20260928 23:12:07.007022 7586 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:07.007084 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:07.008333 7586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:07.008827 7601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:07.008980 7586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:07.009057 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "7b1d17ac801c4df7b815e158e52a3c17"
format_stamp: "Formatted at 2026-09-28 23:12:06 on dist-test-slave-6vgd"
server_key: "d6940da070a75ded2d51a5da9eae46c0"
server_key_iv: "61c5d91d0720c47ef7a7899564be4cd6"
server_key_version: "encryptionkey@0"
I20260928 23:12:07.009414 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:07.021265 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:07.022954 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:07.023058 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:07.023228 7586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:07.023490 7586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:07.023520 7586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:07.023542 7586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:07.023566 7586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:07.028705 7586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41905
I20260928 23:12:07.028744 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41905 every 8 connection(s)
I20260928 23:12:07.029074 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:12:07 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 23:12:07 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 23:12:07 dist-test-slave-6vgd krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:07.038576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 7586
I20260928 23:12:07.039319 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.029759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42303 (local address 127.0.98.60:35359)
0928 23:12:07.029809 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.029811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.030210 (+ 399us) server_negotiation.cc:407] Connection header received
0928 23:12:07.031145 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.031146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.031157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.031187 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.031932 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.032688 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.033961 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034260 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034324 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.036397 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.036398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.036398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.036405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.037960 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.038294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.038294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.038295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.038346 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.038617 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.038617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.038618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.038744 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.038785 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.039147 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.039227 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:07.039808 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.029743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35359 (local address 127.0.98.3:42303)
0928 23:12:07.030138 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.030148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.031036 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.031294 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.031299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.031624 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:07.031850 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.031854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.033268 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.033269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.033882 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.033884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034329 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.034397 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.034963 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:07.034972 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:07.036313 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:07.038017 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.038023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.038026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.038233 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.038399 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.038399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.038400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.038539 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.038832 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:07.038840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:07.038993 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.039439 (+ 446us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.039606 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260928 23:12:07.039930 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.029829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56335 (local address 127.0.98.62:35539)
0928 23:12:07.029877 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.029879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.030048 (+ 169us) server_negotiation.cc:407] Connection header received
0928 23:12:07.031156 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.031157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.031166 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.031212 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.031954 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.032457 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.033985 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034171 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034202 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.037334 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.037336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.037336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.037344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.038832 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039180 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.039181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.039181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.039227 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039429 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.039429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.039430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.039547 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.039585 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.039767 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.039843 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:12:07.040154 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.030102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60743 (local address 127.0.98.61:46337)
0928 23:12:07.030147 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.030149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.030201 (+ 52us) server_negotiation.cc:407] Connection header received
0928 23:12:07.031262 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.031264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.031273 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.031320 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.031960 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.032693 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.033990 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034269 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034324 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.037601 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.037602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.037602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.037609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.039087 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039424 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.039425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.039426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.039473 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039648 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.039649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.039650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.039779 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.039817 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.039994 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.040067 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:12:07.040249 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.029743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:56335)
0928 23:12:07.029926 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.029968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.031047 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.031295 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.031299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.031635 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:07.031852 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.031857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.032552 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.032553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.033886 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.033889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034276 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034276 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.034397 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.034985 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:07.034989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:07.037253 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:07.038920 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.038921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.038922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.039106 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.039309 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.039310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.039359 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.039627 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:07.039627 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:07.039665 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.040030 (+ 365us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.040116 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":71,"thread_start_us":109,"threads_started":1}
I20260928 23:12:07.040380 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.030030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46337 (local address 127.0.98.3:60743)
0928 23:12:07.030149 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.030157 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.031047 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.031337 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.031338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.031651 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:07.031852 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.031857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.032778 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.032779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.033886 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.033889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.034614 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.034614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.034667 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.035122 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:07.035125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:07.037253 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:07.039161 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.039162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.039348 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.039538 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:07.039538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:07.039539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:07.039585 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:07.039859 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:07.039860 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:07.039896 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.040223 (+ 327us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.040301 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":8,"thread_start_us":65,"threads_started":1}
I20260928 23:12:07.040745 7715 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35359
I20260928 23:12:07.040753 7717 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46337
I20260928 23:12:07.040859 7715 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:07.040861 7717 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:07.040935 7716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35539
I20260928 23:12:07.040966 7716 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:07.041059 7715 heartbeater.cc:511] Master 127.0.98.60:35359 requested a full tablet report, sending...
I20260928 23:12:07.041054 7716 heartbeater.cc:511] Master 127.0.98.62:35539 requested a full tablet report, sending...
I20260928 23:12:07.041054 7717 heartbeater.cc:511] Master 127.0.98.61:46337 requested a full tablet report, sending...
I20260928 23:12:07.041517 6479 ts_manager.cc:194] Re-registered known tserver with Master: 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905)
I20260928 23:12:07.041637 7076 ts_manager.cc:194] Registered new tserver with Master: 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905)
I20260928 23:12:07.041689 7263 ts_manager.cc:194] Registered new tserver with Master: 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905)
I20260928 23:12:07.041970 6479 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56335
I20260928 23:12:07.048166 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.041018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.0.98.60:35359)
0928 23:12:07.041051 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.041052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.041437 (+ 385us) server_negotiation.cc:407] Connection header received
0928 23:12:07.041956 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.041957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.041966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.042013 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.042629 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.043072 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044076 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.044275 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044321 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.045870 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.045871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.045872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.045877 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.047178 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047542 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.047593 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047748 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.047850 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.047929 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.048021 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.048085 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:12:07.048365 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.040907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.0.98.61:46337)
0928 23:12:07.041136 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.041138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.041652 (+ 514us) server_negotiation.cc:407] Connection header received
0928 23:12:07.041734 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.041735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.041745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.041771 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.042533 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.043026 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044029 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.044306 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044367 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.045800 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.045802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.045802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.045809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.047158 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047602 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.047652 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047841 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.047952 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.048014 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.048215 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.048284 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260928 23:12:07.048426 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.041201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.0.98.62:35539)
0928 23:12:07.041270 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.041271 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.042044 (+ 773us) server_negotiation.cc:407] Connection header received
0928 23:12:07.042128 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.042141 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.042151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.042177 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.043404 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.043844 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044733 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.044931 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.044981 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.046068 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.046069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.046070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.046076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.047404 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047734 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.047778 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.047951 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.047952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.047952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.048057 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.048116 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.048292 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.048353 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:12:07.049873 6479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:07.051362 6479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:07.057850 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41905 (local address 127.0.0.1:41218)
0928 23:12:07.054504 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.054515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.054550 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.055003 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.055004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.055006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:07.055046 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.055048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.055922 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.055923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.056738 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.056740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.057648 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.057649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.057701 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.057704 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.057742 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.057785 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:12:07.058133 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.0.98.3:41905)
0928 23:12:07.054831 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.054835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.054846 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:12:07.054883 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.054885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.054900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.055113 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:07.055168 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.055853 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.056827 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.057562 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.057653 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.057985 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.058015 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.058066 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260928 23:12:07.058619 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38425 (local address 127.0.0.1:43236)
0928 23:12:07.054668 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.054676 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.054715 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.054990 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.054991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.054993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:07.055039 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.055041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.056683 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.056684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.057566 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.057568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.058421 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.058422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.058483 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.058487 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.058515 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.058551 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260928 23:12:07.058696 7649 tablet_service.cc:1467] Processing CreateTablet for tablet 99464816d6f3472a811c5eb80960bb28 (DEFAULT_TABLE table=test-table [id=855ba2af646040b4a3b59d65e053f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:07.058861 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41953 (local address 127.0.0.1:59764)
0928 23:12:07.054847 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.054856 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.054906 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.055162 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.055163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.055165 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:07.055198 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.055199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.056476 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.056477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.057532 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.057535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.058629 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.058630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.058697 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.058702 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.058759 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.058802 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":85,"threads_started":1}
I20260928 23:12:07.058897 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.0.98.1:41953)
0928 23:12:07.054865 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.054869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.054898 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:12:07.055014 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.055016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.055031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.055100 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:07.055284 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.056392 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.057625 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.058546 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.058618 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.058689 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.058779 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.058821 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":65,"threads_started":1}
I20260928 23:12:07.058983 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.054696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.98.2:38425)
0928 23:12:07.054831 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.054835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.054846 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:12:07.054883 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.054885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.054901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.055107 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:07.055161 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.056615 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.057651 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.058326 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.058771 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.058853 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.058875 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.058914 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260928 23:12:07.059057 7649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99464816d6f3472a811c5eb80960bb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:07.059476 7376 tablet_service.cc:1467] Processing CreateTablet for tablet 99464816d6f3472a811c5eb80960bb28 (DEFAULT_TABLE table=test-table [id=855ba2af646040b4a3b59d65e053f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:07.059546 7513 tablet_service.cc:1467] Processing CreateTablet for tablet 99464816d6f3472a811c5eb80960bb28 (DEFAULT_TABLE table=test-table [id=855ba2af646040b4a3b59d65e053f3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:07.059784 7376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99464816d6f3472a811c5eb80960bb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:07.059809 7513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99464816d6f3472a811c5eb80960bb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:07.062088 7738 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Bootstrap starting.
I20260928 23:12:07.062726 7738 tablet_bootstrap.cc:654] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:07.063000 7738 log.cc:824] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:07.063206 7739 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Bootstrap starting.
I20260928 23:12:07.063725 7741 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Bootstrap starting.
I20260928 23:12:07.063858 7738 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: No bootstrap required, opened a new log
I20260928 23:12:07.063905 7739 tablet_bootstrap.cc:654] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:07.063917 7738 ts_tablet_manager.cc:1397] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:07.064162 7739 log.cc:824] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:07.064294 7741 tablet_bootstrap.cc:654] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:07.064553 7741 log.cc:824] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:07.064822 7739 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: No bootstrap required, opened a new log
I20260928 23:12:07.064891 7739 ts_tablet_manager.cc:1397] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:07.065306 7741 tablet_bootstrap.cc:492] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: No bootstrap required, opened a new log
I20260928 23:12:07.065460 7741 ts_tablet_manager.cc:1397] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:07.066316 7738 raft_consensus.cc:348] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.066478 7738 raft_consensus.cc:374] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:07.066519 7738 raft_consensus.cc:729] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1d17ac801c4df7b815e158e52a3c17, State: Initialized, Role: FOLLOWER
I20260928 23:12:07.066627 7738 consensus_queue.cc:260] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.066888 7738 ts_tablet_manager.cc:1428] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:12:07.066978 7716 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260928 23:12:07.067333 7739 raft_consensus.cc:348] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.067487 7739 raft_consensus.cc:374] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:07.067523 7739 raft_consensus.cc:729] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d060f45c3d4d3bb655319ccf49d21f, State: Initialized, Role: FOLLOWER
I20260928 23:12:07.067643 7739 consensus_queue.cc:260] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.067929 7739 ts_tablet_manager.cc:1428] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:12:07.068078 7443 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260928 23:12:07.068529 7741 raft_consensus.cc:348] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.068672 7741 raft_consensus.cc:374] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:07.068715 7741 raft_consensus.cc:729] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c35975b75b54ff7a0591a43205d3a54, State: Initialized, Role: FOLLOWER
I20260928 23:12:07.068828 7741 consensus_queue.cc:260] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.069072 7741 ts_tablet_manager.cc:1428] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 23:12:07.069177 7580 heartbeater.cc:503] Master 127.0.98.62:35539 was elected leader, sending a full tablet report...
I20260928 23:12:07.081302 7745 raft_consensus.cc:482] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:07.081421 7745 raft_consensus.cc:504] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.081761 7745 leader_election.cc:288] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905), 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
I20260928 23:12:07.084273 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.081975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59251 (local address 127.0.98.3:41905)
0928 23:12:07.082021 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.082023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.082031 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:07.082078 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.082079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.082090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.082133 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:07.082283 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.082736 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.083633 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.083981 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.084038 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.084047 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.084177 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.084212 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:12:07.084270 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.082001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35793 (local address 127.0.98.2:38425)
0928 23:12:07.082046 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.082048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.082056 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:07.082078 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.082079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.082090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.082138 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:07.082280 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.082731 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.083625 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.083981 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.084038 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.084047 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.084170 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.084202 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:12:07.084290 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.081914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38425 (local address 127.0.98.1:35793)
0928 23:12:07.081965 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.081977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.082013 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.082148 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.082148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.082150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:07.082222 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.082223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.082795 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.082796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.083547 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.083549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.084046 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.084046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.084110 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.084113 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.084158 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.084201 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":35}
I20260928 23:12:07.084506 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.081933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41905 (local address 127.0.98.1:59251)
0928 23:12:07.081975 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:07.081986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:07.082025 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:07.082147 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:07.082148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:07.082149 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:07.082222 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.082223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.082804 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.082805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.083549 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:07.083550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.084052 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.084053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:07.084112 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.084115 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:07.084159 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:12:07.084202 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":19}
I20260928 23:12:07.084580 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99464816d6f3472a811c5eb80960bb28" candidate_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35975b75b54ff7a0591a43205d3a54" is_pre_election: true
I20260928 23:12:07.084743 7533 raft_consensus.cc:2454] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d060f45c3d4d3bb655319ccf49d21f in term 0.
I20260928 23:12:07.084776 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99464816d6f3472a811c5eb80960bb28" candidate_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1d17ac801c4df7b815e158e52a3c17" is_pre_election: true
I20260928 23:12:07.084884 7669 raft_consensus.cc:2454] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8d060f45c3d4d3bb655319ccf49d21f in term 0.
I20260928 23:12:07.084949 7330 leader_election.cc:302] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35975b75b54ff7a0591a43205d3a54, a8d060f45c3d4d3bb655319ccf49d21f; no voters:
I20260928 23:12:07.085060 7745 raft_consensus.cc:2781] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:07.085109 7745 raft_consensus.cc:482] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:07.085131 7745 raft_consensus.cc:3037] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:07.085851 7745 raft_consensus.cc:504] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.085960 7745 leader_election.cc:288] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [CANDIDATE]: Term 1 election: Requested vote from peers 7b1d17ac801c4df7b815e158e52a3c17 (127.0.98.3:41905), 2c35975b75b54ff7a0591a43205d3a54 (127.0.98.2:38425)
I20260928 23:12:07.086093 7533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99464816d6f3472a811c5eb80960bb28" candidate_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c35975b75b54ff7a0591a43205d3a54"
I20260928 23:12:07.086093 7669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99464816d6f3472a811c5eb80960bb28" candidate_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b1d17ac801c4df7b815e158e52a3c17"
I20260928 23:12:07.086165 7669 raft_consensus.cc:3037] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:07.086165 7533 raft_consensus.cc:3037] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:07.087007 7533 raft_consensus.cc:2454] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d060f45c3d4d3bb655319ccf49d21f in term 1.
I20260928 23:12:07.087138 7330 leader_election.cc:302] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c35975b75b54ff7a0591a43205d3a54, a8d060f45c3d4d3bb655319ccf49d21f; no voters:
I20260928 23:12:07.087219 7745 raft_consensus.cc:2781] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:07.087374 7745 raft_consensus.cc:686] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 1 LEADER]: Becoming Leader. State: Replica: a8d060f45c3d4d3bb655319ccf49d21f, State: Running, Role: LEADER
I20260928 23:12:07.087386 7669 raft_consensus.cc:2454] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8d060f45c3d4d3bb655319ccf49d21f in term 1.
I20260928 23:12:07.087467 7745 consensus_queue.cc:237] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } }
I20260928 23:12:07.088222 6478 catalog_manager.cc:5495] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f reported cstate change: term changed from 0 to 1, leader changed from <none> to a8d060f45c3d4d3bb655319ccf49d21f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8d060f45c3d4d3bb655319ccf49d21f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41953 } health_report { overall_health: HEALTHY } } }
I20260928 23:12:07.096920 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.095047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.0.98.1:41953)
0928 23:12:07.095093 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.095095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.095101 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:07.095126 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.095126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.095141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.095193 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:07.095323 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.095747 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.096395 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.096562 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.096602 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.096689 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:07.096784 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:07.096833 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.096846 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.096861 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:12:07.098423 7669 raft_consensus.cc:1261] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 1 FOLLOWER]: Refusing update from remote peer a8d060f45c3d4d3bb655319ccf49d21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:07.098434 7533 raft_consensus.cc:1261] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 1 FOLLOWER]: Refusing update from remote peer a8d060f45c3d4d3bb655319ccf49d21f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:07.098716 7747 consensus_queue.cc:1035] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b1d17ac801c4df7b815e158e52a3c17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:07.098820 7745 consensus_queue.cc:1035] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c35975b75b54ff7a0591a43205d3a54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:07.101032 7750 mvcc.cc:205] Tried to move back new op lower bound from 7334449672592891904 to 7334449672550580224. Current Snapshot: MvccSnapshot[applied={T|T < 7334449672592891904}]
I20260928 23:12:07.102053 7751 mvcc.cc:205] Tried to move back new op lower bound from 7334449672592891904 to 7334449672550580224. Current Snapshot: MvccSnapshot[applied={T|T < 7334449672592891904}]
I20260928 23:12:07.103339 6478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:07.103430 6478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:07.103508 7752 mvcc.cc:205] Tried to move back new op lower bound from 7334449672592891904 to 7334449672550580224. Current Snapshot: MvccSnapshot[applied={T|T < 7334449672592891904 or (T in {7334449672592891904})}]
I20260928 23:12:07.104537 6478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81cb14f6bcb1458a834b9e4f35a277d3: Sending DeleteTablet for 3 replicas of tablet 99464816d6f3472a811c5eb80960bb28
I20260928 23:12:07.104813 7376 tablet_service.cc:1514] Processing DeleteTablet for tablet 99464816d6f3472a811c5eb80960bb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59764
I20260928 23:12:07.104820 7649 tablet_service.cc:1514] Processing DeleteTablet for tablet 99464816d6f3472a811c5eb80960bb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41218
I20260928 23:12:07.104926 7513 tablet_service.cc:1514] Processing DeleteTablet for tablet 99464816d6f3472a811c5eb80960bb28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43236
I20260928 23:12:07.105458 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6447
I20260928 23:12:07.105937 7762 tablet_replica.cc:330] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: stopping tablet replica
I20260928 23:12:07.106040 7762 raft_consensus.cc:2229] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:07.106155 7762 raft_consensus.cc:2258] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:07.106624 7762 ts_tablet_manager.cc:1905] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:07.108466 7762 ts_tablet_manager.cc:1918] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:07.108548 7762 log.cc:1196] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/99464816d6f3472a811c5eb80960bb28
I20260928 23:12:07.108865 7762 ts_tablet_manager.cc:1939] T 99464816d6f3472a811c5eb80960bb28 P 2c35975b75b54ff7a0591a43205d3a54: Deleting consensus metadata
W20260928 23:12:07.112376 7467 connection.cc:535] server connection from 127.0.0.1:43236 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:12:07.112556 7759 tablet_replica.cc:330] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: stopping tablet replica
I20260928 23:12:07.112705 7759 raft_consensus.cc:2229] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:07.112867 7759 raft_consensus.cc:2258] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:07.113231 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.105320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.0.98.60:35359)
0928 23:12:07.105358 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.105360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.105623 (+ 263us) server_negotiation.cc:407] Connection header received
0928 23:12:07.105971 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.105972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.105983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.106009 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.106839 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.107630 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.108643 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.108918 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.108955 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.110474 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.110475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.110475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.110482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.111983 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.112406 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.112408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.112408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.112462 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.112674 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.112675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.112675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.112793 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.112858 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.113045 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.113121 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:12:07.113511 7759 ts_tablet_manager.cc:1905] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:07.115072 7759 ts_tablet_manager.cc:1918] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:07.115183 7759 log.cc:1196] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/99464816d6f3472a811c5eb80960bb28
I20260928 23:12:07.115464 7759 ts_tablet_manager.cc:1939] T 99464816d6f3472a811c5eb80960bb28 P 7b1d17ac801c4df7b815e158e52a3c17: Deleting consensus metadata
W20260928 23:12:07.115824 7605 connection.cc:441] server connection from 127.0.0.1:41218 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41218 (request call id 1) could send its response
I20260928 23:12:07.121807 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.114211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.0.98.60:35359)
0928 23:12:07.114243 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.114245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.114475 (+ 230us) server_negotiation.cc:407] Connection header received
0928 23:12:07.114563 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.114564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.114573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.114599 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.116107 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.116561 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.117454 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.117739 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.117793 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.118983 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.118985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.118985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.118992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.120584 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.120969 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.120970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.120971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.121019 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.121232 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.121233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.121233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.121361 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.121429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.121620 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.121697 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:12:07.121856 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:07.114042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.98.61:46337)
0928 23:12:07.114086 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:07.114088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:07.114476 (+ 388us) server_negotiation.cc:407] Connection header received
0928 23:12:07.114565 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:07.114566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:07.114576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:07.114616 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:07.116105 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.116853 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.117813 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:07.118117 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:07.118174 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:07.119415 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:07.119416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:07.119417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:07.119424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:07.120860 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.121179 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.121180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.121180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.121262 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:07.121414 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:07.121415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:07.121415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:07.121516 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:07.121592 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:07.121690 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:12:07.121760 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:12:07.125304 7760 tablet_replica.cc:330] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: stopping tablet replica
I20260928 23:12:07.125420 7760 raft_consensus.cc:2229] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:07.125533 7760 raft_consensus.cc:2258] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:07.125948 7760 ts_tablet_manager.cc:1905] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:07.127537 7760 ts_tablet_manager.cc:1918] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:07.127605 7760 log.cc:1196] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/99464816d6f3472a811c5eb80960bb28
I20260928 23:12:07.127861 7760 ts_tablet_manager.cc:1939] T 99464816d6f3472a811c5eb80960bb28 P a8d060f45c3d4d3bb655319ccf49d21f: Deleting consensus metadata
W20260928 23:12:07.128175 7331 connection.cc:441] server connection from 127.0.0.1:59764 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59764 (request call id 1) could send its response
I20260928 23:12:08.102576 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:08.102294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:38931)
0928 23:12:08.102440 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:08.102517 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
W20260928 23:12:08.102918 7580 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:08.103324 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:08.103118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:57399)
0928 23:12:08.103222 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:08.103284 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
W20260928 23:12:08.103540 7443 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:08.103924 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:08.103690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:42013)
0928 23:12:08.103803 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:08.103888 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":38,"threads_started":1}
W20260928 23:12:08.104104 7716 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35539 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:08.604662 7773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:08.604755 7773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:08.604946 7773 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:08.605305 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:08.605089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:56078)
0928 23:12:08.605187 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:08.605265 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
W20260928 23:12:08.605440 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:08.605800 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:08.605707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:56084)
0928 23:12:08.605752 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:08.605779 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:12:08.605883 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:08.605922 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:08.605994 7773 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:09.103521 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:09.103276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:45181)
0928 23:12:09.103413 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:09.103467 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260928 23:12:09.104010 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:09.103843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:40909)
0928 23:12:09.103931 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:09.103969 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
I20260928 23:12:09.104568 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:09.104387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:60391)
0928 23:12:09.104500 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:09.104529 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":49,"threads_started":1}
I20260928 23:12:10.104295 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:10.103995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:39961)
0928 23:12:10.104128 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:10.104171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 23:12:10.104584 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:10.104392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:47441)
0928 23:12:10.104511 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:10.104543 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
I20260928 23:12:10.105086 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:10.104917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:56951)
0928 23:12:10.105013 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:10.105044 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260928 23:12:10.548273 7782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:10.548368 7782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:10.548496 7782 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:10.548852 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:10.548644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:56348)
0928 23:12:10.548777 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:10.548815 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 23:12:10.549265 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:10.549142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:56362)
0928 23:12:10.549185 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:10.549240 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:12:10.549356 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:10.549407 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:10.549479 7782 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:11.105020 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:11.104713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:34595)
0928 23:12:11.104924 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:11.104966 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260928 23:12:11.105257 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:11.104978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:42183)
0928 23:12:11.105080 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:11.105117 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
I20260928 23:12:11.105620 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:11.105428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:54013)
0928 23:12:11.105541 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:11.105574 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260928 23:12:12.105742 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:12.105457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:42425)
0928 23:12:12.105641 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:12.105686 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260928 23:12:12.105957 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:12.105718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:51829)
0928 23:12:12.105887 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:12.105917 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":45,"threads_started":1}
I20260928 23:12:12.106205 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:12.106036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:39575)
0928 23:12:12.106136 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:12.106165 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":44,"threads_started":1}
I20260928 23:12:13.106535 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:13.106288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:36635)
0928 23:12:13.106448 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:13.106481 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":64,"threads_started":1}
I20260928 23:12:13.106652 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:13.106193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:33987)
0928 23:12:13.106313 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:13.106350 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260928 23:12:13.106791 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:13.106522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:42011)
0928 23:12:13.106716 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:13.106752 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":46,"threads_started":1}
I20260928 23:12:13.421725 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:13.421821 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:13.421962 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:13.422292 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:13.422114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:54808)
0928 23:12:13.422219 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:13.422254 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260928 23:12:13.422660 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:13.422576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:54818)
0928 23:12:13.422618 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:13.422641 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:12:13.422729 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:13.422761 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:13.422868 7794 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:14.107322 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:14.107054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:54941)
0928 23:12:14.107228 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:14.107268 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":59,"threads_started":1}
I20260928 23:12:14.107409 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:14.107112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:54303)
0928 23:12:14.107325 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:14.107360 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":50,"threads_started":1}
I20260928 23:12:14.107440 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:14.107290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:42267)
0928 23:12:14.107377 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:14.107402 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":42,"threads_started":1}
I20260928 23:12:15.107940 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:15.107737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:54905)
0928 23:12:15.107853 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:15.107885 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260928 23:12:15.108001 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:15.107819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:43761)
0928 23:12:15.107925 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:15.107957 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260928 23:12:15.108033 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:15.107884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:40699)
0928 23:12:15.107968 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:15.107997 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":47,"threads_started":1}
I20260928 23:12:16.108506 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:16.108284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:52253)
0928 23:12:16.108413 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:16.108454 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260928 23:12:16.108506 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:16.108313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:40915)
0928 23:12:16.108424 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:16.108454 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
I20260928 23:12:16.108587 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:16.108362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:43269)
0928 23:12:16.108504 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:16.108540 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
I20260928 23:12:17.109110 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:17.108890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:49957)
0928 23:12:17.109015 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:17.109052 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 23:12:17.109138 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:17.108897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:42329)
0928 23:12:17.109065 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:17.109099 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":54,"threads_started":1}
I20260928 23:12:17.109164 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:17.109019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:53151)
0928 23:12:17.109105 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:17.109130 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":41,"threads_started":1}
I20260928 23:12:17.660107 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:17.660202 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:17.660353 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:17.660692 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:17.660501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55154)
0928 23:12:17.660613 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:17.660654 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
W20260928 23:12:17.660779 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 23:12:17.661127 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:17.661034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:55170)
0928 23:12:17.661077 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:17.661106 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:12:17.661202 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:17.661274 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:17.661350 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:18.109740 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:18.109522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:44659)
0928 23:12:18.109659 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:18.109697 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260928 23:12:18.109745 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:18.109570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:44237)
0928 23:12:18.109678 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:18.109702 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":42,"threads_started":1}
I20260928 23:12:18.109745 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:18.109525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:44697)
0928 23:12:18.109665 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:18.109700 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260928 23:12:19.110304 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:19.110075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:60803)
0928 23:12:19.110238 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:19.110265 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":60,"threads_started":1}
I20260928 23:12:19.110304 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:19.110075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:56967)
0928 23:12:19.110208 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:19.110253 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 23:12:19.110443 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:19.110075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:57679)
0928 23:12:19.110302 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:19.110340 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
I20260928 23:12:20.110900 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:20.110656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:44295)
0928 23:12:20.110782 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:20.110823 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 23:12:20.111054 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:20.110884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:33265)
0928 23:12:20.110976 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:20.111008 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":40,"threads_started":1}
I20260928 23:12:20.111171 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:20.110755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:45953)
0928 23:12:20.111096 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:20.111131 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":54,"threads_started":1}
I20260928 23:12:21.111701 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:21.111483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:48391)
0928 23:12:21.111609 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:21.111647 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":52,"threads_started":1}
I20260928 23:12:21.111706 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:21.111421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:32975)
0928 23:12:21.111643 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:21.111671 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":59,"threads_started":1}
I20260928 23:12:21.111807 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:21.111618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:44321)
0928 23:12:21.111729 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:21.111763 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260928 23:12:22.112383 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:22.112166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:37637)
0928 23:12:22.112290 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:22.112331 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":51,"threads_started":1}
I20260928 23:12:22.112385 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:22.112088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:43311)
0928 23:12:22.112296 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:22.112333 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":59,"threads_started":1}
I20260928 23:12:22.112430 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:22.112178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:43651)
0928 23:12:22.112365 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:22.112394 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":42,"threads_started":1}
I20260928 23:12:22.656466 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:22.656579 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:22.656702 7827 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:22.657076 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:22.656873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41386)
0928 23:12:22.656986 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:22.657029 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260928 23:12:22.657444 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:22.657374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41394)
0928 23:12:22.657408 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:22.657430 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:12:22.657514 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:22.657588 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:22.657649 7827 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:23.113116 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:23.112877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:44387)
0928 23:12:23.113038 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:23.113071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":54,"threads_started":1}
I20260928 23:12:23.113116 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:23.112952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:56087)
0928 23:12:23.113041 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:23.113071 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":42,"threads_started":1}
I20260928 23:12:23.113099 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:23.112830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:36095)
0928 23:12:23.112998 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:23.113042 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260928 23:12:24.113711 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:24.113476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:34875)
0928 23:12:24.113617 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:24.113660 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260928 23:12:24.113711 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:24.113476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:38383)
0928 23:12:24.113627 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:24.113659 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260928 23:12:24.113802 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:24.113508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:51013)
0928 23:12:24.113720 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:24.113756 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":50,"threads_started":1}
I20260928 23:12:25.114323 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:25.114149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:55701)
0928 23:12:25.114230 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:25.114269 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":39,"threads_started":1}
I20260928 23:12:25.114372 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:25.114176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:51005)
0928 23:12:25.114290 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:25.114324 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260928 23:12:25.114619 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:25.114078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:48473)
0928 23:12:25.114547 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:25.114581 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":78,"threads_started":1}
I20260928 23:12:25.305270 7839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:25.305378 7839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:25.305546 7839 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:25.305904 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:25.305715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41498)
0928 23:12:25.305828 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:25.305864 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260928 23:12:25.305984 7063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 23:12:25.306317 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:25.306231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41510)
0928 23:12:25.306274 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:25.306298 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:12:25.306380 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:25.306416 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:25.306476 7839 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:26.114986 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:26.114765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:43343)
0928 23:12:26.114893 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:26.114932 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 23:12:26.115136 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:26.114977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:52269)
0928 23:12:26.115051 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:26.115092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61,"thread_start_us":43,"threads_started":1}
I20260928 23:12:26.115136 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:26.114821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:37577)
0928 23:12:26.115050 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:26.115092 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":43,"threads_started":1}
I20260928 23:12:27.115610 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:27.115384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:48055)
0928 23:12:27.115508 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:27.115555 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260928 23:12:27.115746 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:27.115570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:59993)
0928 23:12:27.115681 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:27.115711 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260928 23:12:27.115681 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:27.115511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:45299)
0928 23:12:27.115602 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:27.115634 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":38,"threads_started":1}
I20260928 23:12:28.116261 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:28.116012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:44361)
0928 23:12:28.116155 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:28.116210 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260928 23:12:28.116292 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:28.116097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:47775)
0928 23:12:28.116209 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:28.116246 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260928 23:12:28.116397 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:28.116211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:52823)
0928 23:12:28.116333 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:28.116362 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 23:12:29.116869 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:29.116646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:37719)
0928 23:12:29.116770 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:29.116813 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260928 23:12:29.116947 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:29.116746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:37923)
0928 23:12:29.116843 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:29.116899 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":39,"threads_started":1}
I20260928 23:12:29.117082 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:29.116794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:57465)
0928 23:12:29.117014 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:29.117044 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":53,"threads_started":1}
I20260928 23:12:29.263007 7854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 81cb14f6bcb1458a834b9e4f35a277d3)
I20260928 23:12:29.263110 7854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "81cb14f6bcb1458a834b9e4f35a277d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35539 } } peers { permanent_uuid: "287a0334a837431a8a526aa61539ac05" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46337 } attrs { promote: false } } peers { permanent_uuid: "a3b2c868b32145d68bd0d861457885d6" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35359 } attrs { promote: true } }
I20260928 23:12:29.263238 7854 leader_election.cc:288] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539)
I20260928 23:12:29.263571 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:29.263382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41692)
0928 23:12:29.263486 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:29.263531 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260928 23:12:29.263932 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:29.263843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.0.1:41702)
0928 23:12:29.263884 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:29.263908 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:12:29.264014 7063 leader_election.cc:334] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 81cb14f6bcb1458a834b9e4f35a277d3 (127.0.98.62:35539): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
I20260928 23:12:29.264065 7063 leader_election.cc:302] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 287a0334a837431a8a526aa61539ac05; no voters: 81cb14f6bcb1458a834b9e4f35a277d3
I20260928 23:12:29.264135 7854 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 287a0334a837431a8a526aa61539ac05 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:12:30.117628 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:30.117369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:36367)
0928 23:12:30.117526 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:30.117574 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":50,"threads_started":1}
I20260928 23:12:30.117645 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:30.117348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:55671)
0928 23:12:30.117485 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:30.117596 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260928 23:12:30.117653 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:30.117348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:53201)
0928 23:12:30.117582 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:30.117610 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260928 23:12:31.118196 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:31.117977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:46235)
0928 23:12:31.118114 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:31.118150 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260928 23:12:31.118237 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:31.118056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:49859)
0928 23:12:31.118158 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:31.118191 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
I20260928 23:12:31.118391 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:31.118120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:39515)
0928 23:12:31.118318 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:31.118352 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":40,"threads_started":1}
I20260928 23:12:32.118837 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:32.118596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:46187)
0928 23:12:32.118740 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:32.118782 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 23:12:32.118954 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:32.118743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:40975)
0928 23:12:32.118872 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:32.118905 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 23:12:32.119024 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:32.118851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:45787)
0928 23:12:32.118961 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:32.118987 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260928 23:12:33.119583 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:33.119334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:49181)
0928 23:12:33.119495 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:33.119533 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260928 23:12:33.119580 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:33.119374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:39713)
0928 23:12:33.119474 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:33.119518 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260928 23:12:33.119616 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:33.119355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:33193)
0928 23:12:33.119552 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:33.119583 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":64,"threads_started":1}
I20260928 23:12:34.120250 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:34.119957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:53339)
0928 23:12:34.120166 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:34.120200 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":54,"threads_started":1}
I20260928 23:12:34.120250 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:34.119950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:59741)
0928 23:12:34.120156 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:34.120200 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":59,"threads_started":1}
I20260928 23:12:34.120373 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:34.120156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:46063)
0928 23:12:34.120253 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:34.120284 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":43,"threads_started":1}
I20260928 23:12:35.120961 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:35.120765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:54671)
0928 23:12:35.120883 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:35.120911 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260928 23:12:35.120961 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:35.120686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:59145)
0928 23:12:35.120865 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:35.120908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":76,"threads_started":1}
I20260928 23:12:35.120976 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:35.120765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:59291)
0928 23:12:35.120893 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:35.120928 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260928 23:12:36.121678 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:36.121398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:38045)
0928 23:12:36.121575 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:36.121619 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
I20260928 23:12:36.121726 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:36.121447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:36777)
0928 23:12:36.121661 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:36.121690 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":56,"threads_started":1}
I20260928 23:12:36.121744 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:36.121448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:44731)
0928 23:12:36.121655 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:36.121695 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":54,"threads_started":1}
I20260928 23:12:37.122393 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:37.122195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.2:55055)
0928 23:12:37.122311 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:37.122347 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260928 23:12:37.122535 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:37.122307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.3:60641)
0928 23:12:37.122463 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:37.122492 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":37,"threads_started":1}
I20260928 23:12:37.122584 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:37.122287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35539 (local address 127.0.98.1:59723)
0928 23:12:37.122460 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:37.122492 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":50,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:35539,127.0.98.61:46337,127.0.98.60:35359, num_attempts: 1)
I20260928 23:12:37.257032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 7223
I20260928 23:12:37.262115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 7450
I20260928 23:12:37.265974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 7586
I20260928 23:12:37.269528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 6943
I20260928 23:12:37.272969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 7130
2026-09-28T23:12:37Z chronyd exiting
I20260928 23:12:37.287662 392 test_util.cc:175] -----------------------------------------------
I20260928 23:12:37.287707 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790637099107577-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33265 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:37 dist-test-slave-6vgd krb5kdc[7883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:37 dist-test-slave-6vgd krb5kdc[7883](info): set up 2 sockets
Sep 28 23:12:37 dist-test-slave-6vgd krb5kdc[7883](info): commencing operation
krb5kdc: starting...
W20260928 23:12:38.225294 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 23:12:38.335837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:38.414804 7896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:38.415007 7896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:38.415035 7896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:38.416177 7896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:38.416213 7896 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 23:12:38.416230 7896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:38.416245 7896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:38.416261 7896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:38.416285 7896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:38.417762 7896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46609
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:38.417997 7896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:38.418196 7896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:38.423110 7901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.423121 7902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.423121 7904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.423297 7896 server_base.cc:1034] running on GCE node
I20260928 23:12:38.423532 7896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:38.423816 7896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:38.424954 7896 hybrid_clock.cc:648] HybridClock initialized: now 1790637158424939 us; error 33 us; skew 500 ppm
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:38.427287 7896 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 23:12:38.427675 7896 webserver.cc:460] Webserver started at http://127.0.98.62:40905/ using document root <none> and password file <none>
I20260928 23:12:38.427840 7896 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:38.427877 7896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:38.427963 7896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:38.428918 7896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "60bd2b13a0ec42359e35f6e5f25b9288"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "624d6b96b29b94a20c00406e00fe6c86"
server_key_iv: "f0719b973a97a803ac3ab073f29c8895"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.429234 7896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "60bd2b13a0ec42359e35f6e5f25b9288"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "624d6b96b29b94a20c00406e00fe6c86"
server_key_iv: "f0719b973a97a803ac3ab073f29c8895"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.430364 7896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:38.431047 7911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.431191 7896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:38.431268 7896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "60bd2b13a0ec42359e35f6e5f25b9288"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "624d6b96b29b94a20c00406e00fe6c86"
server_key_iv: "f0719b973a97a803ac3ab073f29c8895"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.431329 7896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:38.452540 7896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:38.454310 7896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:38.454428 7896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:38.458587 7896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46609
I20260928 23:12:38.458665 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46609 every 8 connection(s)
I20260928 23:12:38.458942 7896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:38.459537 7974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:38.460103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 7896
I20260928 23:12:38.460189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:38.460368 392 external_mini_cluster.cc:1428] Setting key 486741bc98b1be88262a6a442ad446ac
I20260928 23:12:38.462241 7974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Bootstrap starting.
I20260928 23:12:38.462898 7974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:38.463193 7974 log.cc:824] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:38.463871 7974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: No bootstrap required, opened a new log
I20260928 23:12:38.465561 7974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } }
I20260928 23:12:38.465715 7974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:38.465761 7974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bd2b13a0ec42359e35f6e5f25b9288, State: Initialized, Role: FOLLOWER
I20260928 23:12:38.465878 7974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } }
I20260928 23:12:38.465953 7974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:38.465992 7974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:38.466034 7974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:38.466471 7974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } }
I20260928 23:12:38.466554 7974 leader_election.cc:302] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60bd2b13a0ec42359e35f6e5f25b9288; no voters:
I20260928 23:12:38.466704 7974 leader_election.cc:288] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:38.466744 7979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:38.466903 7979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [term 1 LEADER]: Becoming Leader. State: Replica: 60bd2b13a0ec42359e35f6e5f25b9288, State: Running, Role: LEADER
I20260928 23:12:38.466946 7974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:38.466990 7979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } }
I20260928 23:12:38.467340 7979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60bd2b13a0ec42359e35f6e5f25b9288. Latest consensus state: current_term: 1 leader_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } } }
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:38.467506 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bd2b13a0ec42359e35f6e5f25b9288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46609 } } }
I20260928 23:12:38.467689 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:38.467568 7979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288 [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:38.468086 7988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:38.468554 7988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:38.470347 7988 catalog_manager.cc:1347] Generated new cluster ID: bd7ffc1dfd4840dbb31ea29d2181346e
I20260928 23:12:38.470429 7988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:38.471342 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.460822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.98.62:46609)
0928 23:12:38.460991 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.461000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.461032 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:12:38.461564 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.461572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.461642 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.461783 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.462321 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.463599 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.464330 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.465432 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.465541 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.468096 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.468100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.468100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.468111 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.469871 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.470266 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.470268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.470268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.470345 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.470552 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.470553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.470553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.470776 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.470898 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.471024 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.471148 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 23:12:38.492077 7988 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:38.492599 7988 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:38.496513 7988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Generated new TSK 0
I20260928 23:12:38.496683 7988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 23:12:38.522194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46609
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:38.599164 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:38.599347 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:38.599367 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:38.600457 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:38.600495 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:38.600517 8000 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:12:38.600546 8000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:38.602029 8000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:38.602240 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:38.602437 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:38.607088 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.607082 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.607084 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.607096 8000 server_base.cc:1034] running on GCE node
I20260928 23:12:38.607388 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:38.607569 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:38.608701 8000 hybrid_clock.cc:648] HybridClock initialized: now 1790637158608692 us; error 26 us; skew 500 ppm
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:38.610918 8000 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:12:38.611299 8000 webserver.cc:460] Webserver started at http://127.0.98.1:40727/ using document root <none> and password file <none>
I20260928 23:12:38.611479 8000 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:38.611533 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:38.611651 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:38.612623 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "35f17c553255e8707ac59b7c6a825ca2"
server_key_iv: "a0f5613e997152e9e0b90beaad942711"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.612929 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "35f17c553255e8707ac59b7c6a825ca2"
server_key_iv: "a0f5613e997152e9e0b90beaad942711"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.614171 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:38.614778 8015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.614925 8000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:38.615000 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "35f17c553255e8707ac59b7c6a825ca2"
server_key_iv: "a0f5613e997152e9e0b90beaad942711"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.615061 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:38.629310 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:38.631089 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:38.631217 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:38.631685 8000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:38.631727 8000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.631754 8000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:38.631786 8000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:38.638159 8000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35089
I20260928 23:12:38.638243 8130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35089 every 8 connection(s)
I20260928 23:12:38.638587 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:38.640848 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.632092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36529 (local address 127.0.98.62:46609)
0928 23:12:38.632153 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.632155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.632326 (+ 171us) server_negotiation.cc:407] Connection header received
0928 23:12:38.632798 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.632800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.632812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.632887 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.633642 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.634422 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.635775 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.636080 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.636150 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.638627 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.638629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.638630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.638638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.639854 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.640186 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.640186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.640187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.640229 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.640382 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.640382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.640382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.640474 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.640515 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.640727 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.640788 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:12:38.640996 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.632053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.1:36529)
0928 23:12:38.632205 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.632244 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.632717 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.632956 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.632960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.633292 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.633537 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.633543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.634507 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.634508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.635683 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.635685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.636192 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.636193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.636331 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.636813 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.636823 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.638510 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.639918 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.639922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.639924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.640131 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.640264 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.640265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.640265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.640344 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.640514 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.640518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.640632 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.640710 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.640821 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 23:12:38.641466 8131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46609
I20260928 23:12:38.641542 8131 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:38.641708 8131 heartbeater.cc:511] Master 127.0.98.62:46609 requested a full tablet report, sending...
I20260928 23:12:38.642208 7927 ts_manager.cc:194] Registered new tserver with Master: c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089)
I20260928 23:12:38.642869 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36529
I20260928 23:12:38.646395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8000
I20260928 23:12:38.646522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:38.646775 392 external_mini_cluster.cc:1428] Setting key 1fdb567f187fc25a50efb15640a87688
I20260928 23:12:38.647451 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.642450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50247 (local address 127.0.98.62:46609)
0928 23:12:38.642487 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.642489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.642590 (+ 101us) server_negotiation.cc:407] Connection header received
0928 23:12:38.642641 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.642641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.642650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.642693 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.642975 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.643501 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.644181 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.644360 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.644418 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.645107 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.645108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.645109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.645113 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.646378 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.646751 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.646751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.646752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.646800 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.646971 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.646971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.646972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.647104 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.647158 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.647293 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.647363 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:12:38.647467 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.642402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.1:50247)
0928 23:12:38.642524 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.642533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.642589 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.642695 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.642696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.642856 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.642901 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.642902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.643593 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.643594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.644097 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.644098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.644419 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.644420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.644459 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.644791 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.644795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.645037 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.646468 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.646469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.646469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.646675 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.646858 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.646859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.646859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.646921 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.647159 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.647160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.647205 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.647265 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.647365 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 23:12:38.670321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46609
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:38.748826 8140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:38.749044 8140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:38.749078 8140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:38.750238 8140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:38.750276 8140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:38.750296 8140 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:12:38.750313 8140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:38.751744 8140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:38.751940 8140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:38.752140 8140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:38.756958 8145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.757103 8140 server_base.cc:1034] running on GCE node
W20260928 23:12:38.757021 8148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.756956 8146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.757608 8140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:38.757874 8140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:38.759011 8140 hybrid_clock.cc:648] HybridClock initialized: now 1790637158758984 us; error 41 us; skew 500 ppm
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:38.762115 8140 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:12:38.762707 8140 webserver.cc:460] Webserver started at http://127.0.98.2:40453/ using document root <none> and password file <none>
I20260928 23:12:38.762961 8140 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:38.763027 8140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:38.763212 8140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:38.764381 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ca2200f67834c0fb0f43c209c09a829"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c8489a00cac3e55b7257d1e11b594b7d"
server_key_iv: "7fddb7d49b312817f59525948de8328c"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.764722 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ca2200f67834c0fb0f43c209c09a829"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c8489a00cac3e55b7257d1e11b594b7d"
server_key_iv: "7fddb7d49b312817f59525948de8328c"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.766110 8140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:38.766945 8155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.767096 8140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:38.767194 8140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "1ca2200f67834c0fb0f43c209c09a829"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c8489a00cac3e55b7257d1e11b594b7d"
server_key_iv: "7fddb7d49b312817f59525948de8328c"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.767268 8140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:38.789867 8140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:38.791711 8140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:38.791831 8140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:38.792351 8140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:38.792398 8140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.792423 8140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:38.792434 8140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:38.799099 8140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36987
I20260928 23:12:38.799167 8270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36987 every 8 connection(s)
I20260928 23:12:38.799501 8140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:38.801473 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.792869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51305 (local address 127.0.98.62:46609)
0928 23:12:38.792911 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.792913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.793176 (+ 263us) server_negotiation.cc:407] Connection header received
0928 23:12:38.793724 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.793726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.793738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.793788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.794439 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.795005 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.796017 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.796217 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.796271 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.798907 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.798909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.798910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.798917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.800317 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.800701 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.800702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.800703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.800757 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.800939 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.800939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.800939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.801039 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.801097 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.801345 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.801408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:12:38.801640 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.792873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.2:51305)
0928 23:12:38.793059 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.793093 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.793626 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.793857 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.793861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.794163 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.794352 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.794356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.795085 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.795086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.795939 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.795941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.796284 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.796285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.796426 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.796995 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.797005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.798817 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.800390 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.800396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.800398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.800635 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.800813 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.800814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.800814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.800904 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.801090 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.801092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.801204 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.801323 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.801444 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260928 23:12:38.802055 8271 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46609
I20260928 23:12:38.802135 8271 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:38.802313 8271 heartbeater.cc:511] Master 127.0.98.62:46609 requested a full tablet report, sending...
I20260928 23:12:38.802650 7928 ts_manager.cc:194] Registered new tserver with Master: 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987)
I20260928 23:12:38.803021 7928 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51305
I20260928 23:12:38.804790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8140
I20260928 23:12:38.804894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:38.805081 392 external_mini_cluster.cc:1428] Setting key e262b02ae0e9cf71587dfbcb31736157
I20260928 23:12:38.808389 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.803097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40011 (local address 127.0.98.62:46609)
0928 23:12:38.803133 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.803136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.803227 (+ 91us) server_negotiation.cc:407] Connection header received
0928 23:12:38.803278 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.803278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.803288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.803331 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.803632 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.804098 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.804769 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.805035 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.805089 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.805950 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.805951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.805952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.805958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.807291 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.807661 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.807662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.807663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.807721 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.807861 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.807862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.807862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.808021 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.808077 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.808224 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.808303 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:12:38.808400 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.803016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.2:40011)
0928 23:12:38.803163 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.803173 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.803226 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.803334 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.803334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.803494 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.803548 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.803550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.804179 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.804180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.804702 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.804703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.805096 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.805097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.805165 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.805602 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.805606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.805874 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.807385 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.807386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.807386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.807556 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.807775 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.807776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.807776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.807825 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.808083 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.808084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.808133 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.808193 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.808296 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:12:38.829003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46609
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:38.908505 8280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:38.908715 8280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:38.908747 8280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:38.909936 8280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:38.909979 8280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:38.910007 8280 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 23:12:38.910030 8280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:38.911491 8280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:38.911731 8280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:38.911937 8280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:38.916664 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:38.916672 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.916826 8280 server_base.cc:1034] running on GCE node
W20260928 23:12:38.916685 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:38.917073 8280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:38.917354 8280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:38.918485 8280 hybrid_clock.cc:648] HybridClock initialized: now 1790637158918475 us; error 36 us; skew 500 ppm
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:38.920933 8280 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:12:38.921427 8280 webserver.cc:460] Webserver started at http://127.0.98.3:42477/ using document root <none> and password file <none>
I20260928 23:12:38.921643 8280 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:38.921697 8280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:38.921821 8280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:38.922873 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "7057ec54eee14ed89258b92aa3052bbe"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c484fc803ec30247b7996a280eb5449c"
server_key_iv: "c9580ac2e1a887fdc309040d5a36e827"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.923204 8280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7057ec54eee14ed89258b92aa3052bbe"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c484fc803ec30247b7996a280eb5449c"
server_key_iv: "c9580ac2e1a887fdc309040d5a36e827"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.924454 8280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:38.925139 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.925294 8280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:38.925374 8280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "7057ec54eee14ed89258b92aa3052bbe"
format_stamp: "Formatted at 2026-09-28 23:12:38 on dist-test-slave-6vgd"
server_key: "c484fc803ec30247b7996a280eb5449c"
server_key_iv: "c9580ac2e1a887fdc309040d5a36e827"
server_key_version: "encryptionkey@0"
I20260928 23:12:38.925441 8280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:38.945111 8280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:38.947045 8280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:38.947181 8280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:38.947697 8280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:38.947741 8280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:38.947769 8280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:38.947786 8280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:38.954394 8280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33679
I20260928 23:12:38.954720 8280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:38.954707 8410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33679 every 8 connection(s)
I20260928 23:12:38.956609 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.948110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57261 (local address 127.0.98.62:46609)
0928 23:12:38.948183 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.948185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.948343 (+ 158us) server_negotiation.cc:407] Connection header received
0928 23:12:38.948843 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.948845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.948857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.948936 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.949631 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.950384 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.951397 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.951685 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.951763 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.954256 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.954258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.954258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.954267 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.955609 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.955939 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.955939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.955940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.955983 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.956168 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.956168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.956168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.956260 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.956302 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.956496 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.956548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:12:38.956753 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.948079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.3:57261)
0928 23:12:38.948219 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.948259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.948753 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.948999 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.949003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.949337 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.949555 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.949561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.950479 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.950480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.951330 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.951332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.951777 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.951778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.951892 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.952365 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.952375 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.954149 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.955686 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.955690 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.955692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.955886 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.956025 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.956025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.956026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.956126 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.956300 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.956303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.956408 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.956480 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.956586 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260928 23:12:38.957195 8411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46609
I20260928 23:12:38.957300 8411 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:38.957458 8411 heartbeater.cc:511] Master 127.0.98.62:46609 requested a full tablet report, sending...
I20260928 23:12:38.957819 7927 ts_manager.cc:194] Registered new tserver with Master: 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:38.958214 7927 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57261
I20260928 23:12:38.962656 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.958136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.98.3:54721)
0928 23:12:38.958247 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.958258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.958316 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.958414 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.958415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.958589 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:38.958652 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.958653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.959315 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.959315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.959822 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.959823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.960109 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.960109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.960151 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.960439 (+ 288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:38.960443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:38.960682 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:38.961980 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.961981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.961981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.962127 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.962260 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:38.962260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:38.962261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:38.962301 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:38.962448 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:38.962449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:38.962477 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 23:12:38.962521 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:12:38.962586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
I20260928 23:12:38.962667 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.958195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54721 (local address 127.0.98.62:46609)
0928 23:12:38.958234 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.958235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.958304 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:12:38.958360 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.958361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.958370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.958410 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.958720 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.959242 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.959883 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.960054 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.960100 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.960741 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.960742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.960742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.960747 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.961919 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.962178 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.962179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.962179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.962218 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.962334 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.962334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.962334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.962411 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.962451 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.962537 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.962584 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:38.964027 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8280
I20260928 23:12:38.964131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:38.964306 392 external_mini_cluster.cc:1428] Setting key eeaed6aa14e9286d9db34002249f6eb6
I20260928 23:12:38.964948 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:12:38.984475 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:38.979597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58038 (local address 127.0.98.62:46609)
0928 23:12:38.979664 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.979666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.979675 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:38.979701 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.979702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.979710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.979768 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.980276 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.980801 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.981548 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.981760 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.981814 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.984396 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58038: BlockingRecv error: recv got EOF from 127.0.0.1:58038 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 23:12:38 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:38.991154 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.985201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.98.62:46609)
0928 23:12:38.985313 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.985315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.985323 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:38.985343 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.985344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.985352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.985396 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:38.985829 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.986285 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.986911 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.987084 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.987135 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:38.989049 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:38.989051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:38.989051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:38.989057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:38.990227 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.990503 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.990504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.990505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.990554 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:38.990711 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:38.990712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:38.990712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:38.990814 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:38.990899 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:38.990988 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 23:12:38.991056 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 23:12:38.992686 7927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:38.994306 7927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:39.000488 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.0.1:40312)
0928 23:12:38.996718 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.996756 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.996812 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.997326 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.997327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.997329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:38.997497 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.997499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998689 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.998690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.999520 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.999522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000193 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.000276 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000290 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.000361 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.000409 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260928 23:12:39.000581 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.0.1:59786)
0928 23:12:38.996760 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.996772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.996824 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.997340 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.997340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.997342 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:38.997410 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.997412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998256 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.998257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.999445 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.999448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000369 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.000437 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000441 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.000490 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.000525 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":12,"thread_start_us":56,"threads_started":1}
I20260928 23:12:39.000636 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.98.1:35089)
0928 23:12:38.997044 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.997049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.997059 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:12:38.997097 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.997099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.997252 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.997370 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:38.997475 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998185 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.999523 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000248 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000349 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.000506 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.000543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260928 23:12:39.000826 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.0.98.2:36987)
0928 23:12:38.997087 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.997090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.997098 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:38.997130 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.997132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.997145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.997254 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:38.997583 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998618 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.999600 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000120 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000181 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000269 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.000641 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.000685 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":86,"threads_started":1}
I20260928 23:12:39.001456 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.0.98.3:33679)
0928 23:12:38.996997 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:38.997002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:38.997011 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:38.997046 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:38.997057 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:38.997069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:38.997133 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:38.997364 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998480 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.999697 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000361 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000417 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000480 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.001300 (+ 820us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.001334 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
I20260928 23:12:39.001438 8205 tablet_service.cc:1467] Processing CreateTablet for tablet 4012287304524f1cafc1a744ee909c7f (DEFAULT_TABLE table=test-table [id=24fa94da096d405fa3b33aa01c4a310c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.001437 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 4012287304524f1cafc1a744ee909c7f (DEFAULT_TABLE table=test-table [id=24fa94da096d405fa3b33aa01c4a310c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.001806 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4012287304524f1cafc1a744ee909c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.001806 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4012287304524f1cafc1a744ee909c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.005429 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:38.996467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.0.1:53352)
0928 23:12:38.996903 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:38.996911 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:38.996944 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:38.997154 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:38.997157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:38.997159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:38.997256 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.997258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:38.998560 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:38.998561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:38.999626 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:38.999627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.000512 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.000513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.000889 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.000894 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.000940 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.005350 (+ 4410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":57,"threads_started":1}
I20260928 23:12:39.005800 8436 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Bootstrap starting.
I20260928 23:12:39.005928 8437 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Bootstrap starting.
I20260928 23:12:39.006145 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 4012287304524f1cafc1a744ee909c7f (DEFAULT_TABLE table=test-table [id=24fa94da096d405fa3b33aa01c4a310c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.006429 8436 tablet_bootstrap.cc:654] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.006466 8437 tablet_bootstrap.cc:654] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.006474 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4012287304524f1cafc1a744ee909c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.006717 8437 log.cc:824] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:39.006755 8436 log.cc:824] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:39.007330 8437 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: No bootstrap required, opened a new log
I20260928 23:12:39.007381 8437 ts_tablet_manager.cc:1397] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:39.008834 8437 raft_consensus.cc:348] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.008960 8437 raft_consensus.cc:374] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.008980 8437 raft_consensus.cc:729] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.009076 8437 consensus_queue.cc:260] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.009341 8440 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Bootstrap starting.
I20260928 23:12:39.009356 8131 heartbeater.cc:503] Master 127.0.98.62:46609 was elected leader, sending a full tablet report...
I20260928 23:12:39.009353 8437 ts_tablet_manager.cc:1428] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:12:39.009975 8440 tablet_bootstrap.cc:654] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.010283 8440 log.cc:824] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:39.010934 8440 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: No bootstrap required, opened a new log
I20260928 23:12:39.010944 8436 tablet_bootstrap.cc:492] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: No bootstrap required, opened a new log
I20260928 23:12:39.010985 8440 ts_tablet_manager.cc:1397] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:12:39.010994 8436 ts_tablet_manager.cc:1397] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 23:12:39.012404 8440 raft_consensus.cc:348] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.012434 8436 raft_consensus.cc:348] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.012522 8440 raft_consensus.cc:374] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.012543 8440 raft_consensus.cc:729] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.012547 8436 raft_consensus.cc:374] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.012578 8436 raft_consensus.cc:729] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.012639 8440 consensus_queue.cc:260] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.012693 8436 consensus_queue.cc:260] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.012871 8440 ts_tablet_manager.cc:1428] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:39.012903 8411 heartbeater.cc:503] Master 127.0.98.62:46609 was elected leader, sending a full tablet report...
I20260928 23:12:39.012940 8436 ts_tablet_manager.cc:1428] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:12:39.013064 8271 heartbeater.cc:503] Master 127.0.98.62:46609 was elected leader, sending a full tablet report...
I20260928 23:12:39.044102 8443 raft_consensus.cc:482] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:39.044226 8443 raft_consensus.cc:504] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.044531 8443 leader_election.cc:288] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987)
I20260928 23:12:39.047436 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.044713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.98.3:58727)
0928 23:12:39.044785 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.044796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.044835 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.044981 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.044982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.044983 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.045052 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.045054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.045698 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.045699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.046604 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.046606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.047173 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.047174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.047254 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.047258 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.047314 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.047356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 23:12:39.047427 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.044795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58727 (local address 127.0.98.2:36987)
0928 23:12:39.044837 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.044839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.044848 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:39.044912 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.044914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.044928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.044976 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.045122 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.045626 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.046681 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.047100 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.047154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.047163 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.047322 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.047354 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:39.047601 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.044770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40935 (local address 127.0.98.1:35089)
0928 23:12:39.044872 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.044874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.045020 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:12:39.045128 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.045129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.045141 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.045198 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.045348 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.045877 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.046900 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.047292 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.047348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.047358 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.047500 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.047535 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 23:12:39.047825 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.044716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.98.3:40935)
0928 23:12:39.044964 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.044973 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.045045 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.045200 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.045201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.045202 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.045288 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.045290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.045951 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.045952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.046821 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.046823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.047361 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.047362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.047428 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.047432 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.047484 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.047517 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":87,"thread_start_us":108,"threads_started":1}
I20260928 23:12:39.047803 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4012287304524f1cafc1a744ee909c7f" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829" is_pre_election: true
I20260928 23:12:39.047937 8225 raft_consensus.cc:2454] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 0.
I20260928 23:12:39.048134 8296 leader_election.cc:302] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, 7057ec54eee14ed89258b92aa3052bbe; no voters:
I20260928 23:12:39.048133 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4012287304524f1cafc1a744ee909c7f" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" is_pre_election: true
I20260928 23:12:39.048249 8085 raft_consensus.cc:2454] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 0.
I20260928 23:12:39.048264 8443 raft_consensus.cc:2781] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:39.048323 8443 raft_consensus.cc:482] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:39.048352 8443 raft_consensus.cc:3037] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.049021 8443 raft_consensus.cc:504] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.049139 8443 leader_election.cc:288] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 election: Requested vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987)
I20260928 23:12:39.049297 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4012287304524f1cafc1a744ee909c7f" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829"
I20260928 23:12:39.049299 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4012287304524f1cafc1a744ee909c7f" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
I20260928 23:12:39.049361 8225 raft_consensus.cc:3037] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.049362 8085 raft_consensus.cc:3037] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:12:39.049993 8272 tablet.cc:2369] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:12:39.050016 8085 raft_consensus.cc:2454] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 1.
I20260928 23:12:39.050002 8225 raft_consensus.cc:2454] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 1.
I20260928 23:12:39.050451 8297 leader_election.cc:302] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7057ec54eee14ed89258b92aa3052bbe, c96aaf1e1b724c56acdf5ea36baaac74; no voters:
I20260928 23:12:39.050573 8443 raft_consensus.cc:2781] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:39.050724 8443 raft_consensus.cc:686] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 1 LEADER]: Becoming Leader. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Running, Role: LEADER
I20260928 23:12:39.050806 8443 consensus_queue.cc:237] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } }
I20260928 23:12:39.051553 7927 catalog_manager.cc:5495] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7057ec54eee14ed89258b92aa3052bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } health_report { overall_health: UNKNOWN } } }
I20260928 23:12:39.060418 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.058491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.3:33679)
0928 23:12:39.058529 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.058531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.058537 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:39.058607 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.058608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.058647 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.058699 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.058794 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.059220 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.059883 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.060057 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.060102 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.060199 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.060285 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.060336 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.060350 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.060364 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:12:39.061973 8225 raft_consensus.cc:1261] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Refusing update from remote peer 7057ec54eee14ed89258b92aa3052bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.061990 8085 raft_consensus.cc:1261] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Refusing update from remote peer 7057ec54eee14ed89258b92aa3052bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.062282 8446 consensus_queue.cc:1035] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.062394 8443 consensus_queue.cc:1035] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.064512 8449 mvcc.cc:205] Tried to move back new op lower bound from 7334449803515707392 to 7334449803472707584. Current Snapshot: MvccSnapshot[applied={T|T < 7334449803515707392}]
I20260928 23:12:39.064659 8450 mvcc.cc:205] Tried to move back new op lower bound from 7334449803515707392 to 7334449803472707584. Current Snapshot: MvccSnapshot[applied={T|T < 7334449803515707392}]
I20260928 23:12:39.065289 8451 mvcc.cc:205] Tried to move back new op lower bound from 7334449803515707392 to 7334449803472707584. Current Snapshot: MvccSnapshot[applied={T|T < 7334449803515707392 or (T in {7334449803515707392})}]
I20260928 23:12:39.066593 7928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.066674 7928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.067358 7928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Sending DeleteTablet for 3 replicas of tablet 4012287304524f1cafc1a744ee909c7f
I20260928 23:12:39.067703 8065 tablet_service.cc:1514] Processing DeleteTablet for tablet 4012287304524f1cafc1a744ee909c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59786
I20260928 23:12:39.067689 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 4012287304524f1cafc1a744ee909c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53352
I20260928 23:12:39.067762 8205 tablet_service.cc:1514] Processing DeleteTablet for tablet 4012287304524f1cafc1a744ee909c7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40312
I20260928 23:12:39.067907 8458 tablet_replica.cc:330] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: stopping tablet replica
I20260928 23:12:39.067994 8458 raft_consensus.cc:2229] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:39.067997 8459 tablet_replica.cc:330] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: stopping tablet replica
I20260928 23:12:39.068106 8458 raft_consensus.cc:2258] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.068221 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:39.068348 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:39.068491 8459 raft_consensus.cc:2229] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.068533 8458 ts_tablet_manager.cc:1905] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.069500 8459 raft_consensus.cc:2258] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.069624 8205 tablet_service.cc:1467] Processing CreateTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b (DEFAULT_TABLE table=test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.069727 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680317b7b6db4a9da44cd4bb96194c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.070079 8459 ts_tablet_manager.cc:1905] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.070295 8458 ts_tablet_manager.cc:1918] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.070366 8458 log.cc:1196] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/4012287304524f1cafc1a744ee909c7f
I20260928 23:12:39.070595 8458 ts_tablet_manager.cc:1939] T 4012287304524f1cafc1a744ee909c7f P 7057ec54eee14ed89258b92aa3052bbe: Deleting consensus metadata
I20260928 23:12:39.070963 8436 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Bootstrap starting.
I20260928 23:12:39.070998 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b (DEFAULT_TABLE table=test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.071081 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680317b7b6db4a9da44cd4bb96194c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.071372 8436 tablet_bootstrap.cc:654] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.071684 7915 catalog_manager.cc:4841] TS 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679): tablet 4012287304524f1cafc1a744ee909c7f (table test-table [id=24fa94da096d405fa3b33aa01c4a310c]) successfully deleted
I20260928 23:12:39.072034 8459 ts_tablet_manager.cc:1918] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.072077 8436 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: No bootstrap required, opened a new log
I20260928 23:12:39.072116 8436 ts_tablet_manager.cc:1397] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:39.072181 8440 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Bootstrap starting.
I20260928 23:12:39.072203 8459 log.cc:1196] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/4012287304524f1cafc1a744ee909c7f
I20260928 23:12:39.072271 8436 raft_consensus.cc:348] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.072356 8436 raft_consensus.cc:374] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.072386 8436 raft_consensus.cc:729] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.072430 8436 consensus_queue.cc:260] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.072515 8436 ts_tablet_manager.cc:1428] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:39.072573 8440 tablet_bootstrap.cc:654] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.072729 8459 ts_tablet_manager.cc:1939] T 4012287304524f1cafc1a744ee909c7f P c96aaf1e1b724c56acdf5ea36baaac74: Deleting consensus metadata
I20260928 23:12:39.073108 8440 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: No bootstrap required, opened a new log
I20260928 23:12:39.073156 8440 ts_tablet_manager.cc:1397] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:39.073299 8460 tablet_replica.cc:330] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: stopping tablet replica
I20260928 23:12:39.073307 8440 raft_consensus.cc:348] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.073367 8440 raft_consensus.cc:374] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.073361 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b (DEFAULT_TABLE table=test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.073392 8440 raft_consensus.cc:729] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.073400 8460 raft_consensus.cc:2229] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.073441 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680317b7b6db4a9da44cd4bb96194c1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.073427 8440 consensus_queue.cc:260] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.073493 8460 raft_consensus.cc:2258] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.073508 8440 ts_tablet_manager.cc:1428] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.074297 8437 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Bootstrap starting.
I20260928 23:12:39.074491 7913 catalog_manager.cc:4841] TS c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089): tablet 4012287304524f1cafc1a744ee909c7f (table test-table [id=24fa94da096d405fa3b33aa01c4a310c]) successfully deleted
I20260928 23:12:39.074661 8437 tablet_bootstrap.cc:654] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.075256 8437 tablet_bootstrap.cc:492] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: No bootstrap required, opened a new log
I20260928 23:12:39.075299 8437 ts_tablet_manager.cc:1397] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:39.075409 8437 raft_consensus.cc:348] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.075454 8437 raft_consensus.cc:374] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.075466 8437 raft_consensus.cc:729] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.075495 8437 consensus_queue.cc:260] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.075567 8437 ts_tablet_manager.cc:1428] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.075843 8460 ts_tablet_manager.cc:1905] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.077131 8460 ts_tablet_manager.cc:1918] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.077183 8460 log.cc:1196] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/4012287304524f1cafc1a744ee909c7f
I20260928 23:12:39.077406 8460 ts_tablet_manager.cc:1939] T 4012287304524f1cafc1a744ee909c7f P 1ca2200f67834c0fb0f43c209c09a829: Deleting consensus metadata
I20260928 23:12:39.077741 7912 catalog_manager.cc:4841] TS 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987): tablet 4012287304524f1cafc1a744ee909c7f (table test-table [id=24fa94da096d405fa3b33aa01c4a310c]) successfully deleted
W20260928 23:12:39.139268 8132 tablet.cc:2369] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:12:39.146963 8444 raft_consensus.cc:482] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:39.147069 8444 raft_consensus.cc:504] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.147387 8444 leader_election.cc:288] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.150645 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.147759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60091 (local address 127.0.98.3:33679)
0928 23:12:39.147798 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.147800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.147921 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:12:39.147952 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.147953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.147963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.148023 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.148159 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.148813 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.150004 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.150350 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.150408 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.150418 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.150545 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.150577 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:39.150727 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.147541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.2:60091)
0928 23:12:39.147851 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.147860 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.147910 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.148031 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.148031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.148032 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.148101 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.148102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.148899 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.148900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.149934 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.149936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.150416 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.150417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.150486 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.150491 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.150534 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.150577 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":191,"thread_start_us":57,"threads_started":1}
I20260928 23:12:39.150864 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.147682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33173 (local address 127.0.98.1:35089)
0928 23:12:39.147717 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.147718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.147726 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:39.147749 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.147750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.147759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.147800 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.147984 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.148531 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.149799 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.150347 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.150417 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.150428 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.150760 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.150803 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 23:12:39.150732 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.147536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.98.2:33173)
0928 23:12:39.147587 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.147599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.147640 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.147846 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.147847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.147848 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.147916 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.147918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.148605 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.148606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.149715 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.149717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.150461 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.150462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.150545 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.150548 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.150590 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.150634 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:12:39.151190 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680317b7b6db4a9da44cd4bb96194c1b" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe" is_pre_election: true
I20260928 23:12:39.151316 8365 raft_consensus.cc:2454] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 0.
I20260928 23:12:39.151286 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680317b7b6db4a9da44cd4bb96194c1b" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" is_pre_election: true
I20260928 23:12:39.151353 8085 raft_consensus.cc:2454] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 0.
I20260928 23:12:39.151502 8159 leader_election.cc:302] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, 7057ec54eee14ed89258b92aa3052bbe; no voters:
I20260928 23:12:39.151636 8444 raft_consensus.cc:2781] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:39.151674 8444 raft_consensus.cc:482] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:39.151692 8444 raft_consensus.cc:3037] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.152554 8444 raft_consensus.cc:504] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.152686 8444 leader_election.cc:288] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 election: Requested vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.152849 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680317b7b6db4a9da44cd4bb96194c1b" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe"
I20260928 23:12:39.152911 8365 raft_consensus.cc:3037] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.153281 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680317b7b6db4a9da44cd4bb96194c1b" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
I20260928 23:12:39.153366 8085 raft_consensus.cc:3037] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.153666 8365 raft_consensus.cc:2454] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 1.
I20260928 23:12:39.153800 8159 leader_election.cc:302] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, 7057ec54eee14ed89258b92aa3052bbe; no voters:
I20260928 23:12:39.154083 8085 raft_consensus.cc:2454] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 1.
I20260928 23:12:39.154183 8444 raft_consensus.cc:2781] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:39.154291 8444 raft_consensus.cc:686] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Running, Role: LEADER
I20260928 23:12:39.154356 8444 consensus_queue.cc:237] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.155023 7928 catalog_manager.cc:5495] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:39.192076 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.185929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.0.98.62:46609)
0928 23:12:39.186007 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.186009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.186121 (+ 112us) server_negotiation.cc:407] Connection header received
0928 23:12:39.186178 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.186179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.186189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.186236 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:39.186693 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.187176 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.187859 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.188030 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.188083 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.189998 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:39.190000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:39.190000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:39.190007 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:39.191087 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:39.191401 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:39.191402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:39.191402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:39.191448 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:39.191606 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:39.191607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:39.191607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:39.191707 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:39.191761 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.191900 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.191979 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 23:12:39.192154 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.185872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46609 (local address 127.0.0.1:58060)
0928 23:12:39.186027 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.186038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.186126 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.186245 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.186245 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.186563 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:39.186609 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.186612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.187244 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.187245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.187781 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.187783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.188092 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.188092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.188139 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.188624 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:39.188628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:39.189917 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:39.191155 (+ 1238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:39.191162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:39.191165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:39.191341 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:39.191498 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:39.191499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:39.191499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:39.191548 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:39.191766 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:39.191773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:39.191818 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.191874 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.192045 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":72,"thread_start_us":92,"threads_started":1}
I20260928 23:12:39.193874 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 23:12:39.194072 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:39.195380 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 251b4a5ff2904afbb2f204545f39eb65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bed91788b0143a8acf25754bdda1c87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:12:39.195374 8205 tablet_service.cc:1467] Processing CreateTablet for tablet 251b4a5ff2904afbb2f204545f39eb65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bed91788b0143a8acf25754bdda1c87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:12:39.195473 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 251b4a5ff2904afbb2f204545f39eb65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5bed91788b0143a8acf25754bdda1c87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 23:12:39.195531 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251b4a5ff2904afbb2f204545f39eb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.195531 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251b4a5ff2904afbb2f204545f39eb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.195618 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 251b4a5ff2904afbb2f204545f39eb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.196695 8440 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: Bootstrap starting.
I20260928 23:12:39.196940 8436 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: Bootstrap starting.
I20260928 23:12:39.196943 8437 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: Bootstrap starting.
I20260928 23:12:39.197063 8440 tablet_bootstrap.cc:654] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.197293 8436 tablet_bootstrap.cc:654] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.197350 8437 tablet_bootstrap.cc:654] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.197929 8436 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: No bootstrap required, opened a new log
I20260928 23:12:39.197929 8437 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: No bootstrap required, opened a new log
I20260928 23:12:39.197983 8436 ts_tablet_manager.cc:1397] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:39.197983 8437 ts_tablet_manager.cc:1397] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:39.198009 8440 tablet_bootstrap.cc:492] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: No bootstrap required, opened a new log
I20260928 23:12:39.198052 8440 ts_tablet_manager.cc:1397] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:39.198118 8437 raft_consensus.cc:348] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198154 8440 raft_consensus.cc:348] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198199 8437 raft_consensus.cc:374] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.198210 8440 raft_consensus.cc:374] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.198235 8440 raft_consensus.cc:729] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.198235 8437 raft_consensus.cc:729] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.198248 8436 raft_consensus.cc:348] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198275 8437 consensus_queue.cc:260] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198276 8440 consensus_queue.cc:260] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198312 8436 raft_consensus.cc:374] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.198340 8436 raft_consensus.cc:729] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.198361 8440 ts_tablet_manager.cc:1428] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.198364 8437 ts_tablet_manager.cc:1428] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:39.198375 8436 consensus_queue.cc:260] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.198455 8436 ts_tablet_manager.cc:1428] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:39.198477 8446 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.198546 8446 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.198509 8444 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.198585 8444 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.198555 8441 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.198624 8441 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 23:12:39.205382 8412 tablet.cc:2369] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:12:39.420691 8444 raft_consensus.cc:482] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:39.420775 8444 raft_consensus.cc:504] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.420907 8444 leader_election.cc:288] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.421092 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251b4a5ff2904afbb2f204545f39eb65" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" is_pre_election: true
I20260928 23:12:39.421139 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251b4a5ff2904afbb2f204545f39eb65" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe" is_pre_election: true
I20260928 23:12:39.421234 8365 raft_consensus.cc:2454] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 0.
I20260928 23:12:39.421254 8085 raft_consensus.cc:2454] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 0.
I20260928 23:12:39.421383 8159 leader_election.cc:302] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, 7057ec54eee14ed89258b92aa3052bbe; no voters:
I20260928 23:12:39.421509 8444 raft_consensus.cc:2781] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:39.421555 8444 raft_consensus.cc:482] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:39.421568 8444 raft_consensus.cc:3037] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.422194 8444 raft_consensus.cc:504] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.422307 8444 leader_election.cc:288] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 election: Requested vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.422456 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251b4a5ff2904afbb2f204545f39eb65" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
I20260928 23:12:39.422462 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "251b4a5ff2904afbb2f204545f39eb65" candidate_uuid: "1ca2200f67834c0fb0f43c209c09a829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe"
I20260928 23:12:39.422528 8085 raft_consensus.cc:3037] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.422530 8365 raft_consensus.cc:3037] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.423101 8085 raft_consensus.cc:2454] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 1.
I20260928 23:12:39.423175 8365 raft_consensus.cc:2454] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ca2200f67834c0fb0f43c209c09a829 in term 1.
I20260928 23:12:39.423283 8157 leader_election.cc:302] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, c96aaf1e1b724c56acdf5ea36baaac74; no voters:
I20260928 23:12:39.423365 8444 raft_consensus.cc:2781] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:39.423405 8444 raft_consensus.cc:686] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Running, Role: LEADER
I20260928 23:12:39.423448 8444 consensus_queue.cc:237] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.423688 8465 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: TxnStatusTablet state changed. Reason: New leader 1ca2200f67834c0fb0f43c209c09a829. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.423756 8465 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:12:39.423902 8477 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 23:12:39.424012 8477 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 23:12:39.424013 7927 catalog_manager.cc:5495] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
I20260928 23:12:39.480995 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.478328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.0.1:40314)
0928 23:12:39.478394 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.478406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.478453 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.478613 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.478614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.478615 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:12:39.478666 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.478669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.479517 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.479518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.480081 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.480082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.480524 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.480524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.480568 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.480581 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.480863 (+ 282us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.480868 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.480893 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.480933 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:12:39.481009 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.478414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.0.98.2:36987)
0928 23:12:39.478472 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.478474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.478483 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:39.478515 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.478516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.478535 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.478615 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.478724 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.479440 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.480159 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.480451 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.480531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.480683 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.480800 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.480881 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.480906 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.480930 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:12:39.482197 8085 raft_consensus.cc:1261] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Refusing update from remote peer 1ca2200f67834c0fb0f43c209c09a829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.482295 8365 raft_consensus.cc:1261] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Refusing update from remote peer 1ca2200f67834c0fb0f43c209c09a829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.482565 8465 consensus_queue.cc:1035] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.482700 8444 consensus_queue.cc:1035] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.483281 8441 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: TxnStatusTablet state changed. Reason: New leader 1ca2200f67834c0fb0f43c209c09a829. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.483297 8446 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: TxnStatusTablet state changed. Reason: New leader 1ca2200f67834c0fb0f43c209c09a829. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.483363 8441 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.483381 8446 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.484010 8444 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.484088 8444 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:12:39.484529 8441 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.484604 8441 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P c96aaf1e1b724c56acdf5ea36baaac74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.484632 8465 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.484701 8465 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 1ca2200f67834c0fb0f43c209c09a829: This TxnStatusTablet replica's current role is: LEADER
I20260928 23:12:39.485023 8446 tablet_replica.cc:439] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ca2200f67834c0fb0f43c209c09a829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } } }
I20260928 23:12:39.485095 8446 tablet_replica.cc:442] T 251b4a5ff2904afbb2f204545f39eb65 P 7057ec54eee14ed89258b92aa3052bbe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 23:12:39.487782 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.485837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.0.98.2:36987)
0928 23:12:39.485867 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.485869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.485876 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:39.485893 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.485894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.485904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.485948 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.486063 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.486513 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.487204 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.487401 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.487451 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.487572 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.487604 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.487647 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.487703 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.487721 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260928 23:12:39.492508 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.490011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54443 (local address 127.0.98.2:36987)
0928 23:12:39.490047 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.490049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.490189 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:12:39.490247 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.490247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.490254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.490293 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.490417 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.490836 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.491919 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.492085 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.492126 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.492311 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.492332 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.492361 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.492431 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.492446 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260928 23:12:39.492600 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.489954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.98.2:54443)
0928 23:12:39.490119 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.490130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.490194 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.490300 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.490300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.490302 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:12:39.490366 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.490367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.490906 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.490907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.491835 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.491838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.492149 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.492150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.492221 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.492236 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.492378 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.492380 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.492429 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.492472 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260928 23:12:39.494406 8365 raft_consensus.cc:1261] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Refusing update from remote peer 1ca2200f67834c0fb0f43c209c09a829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.494482 8085 raft_consensus.cc:1261] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Refusing update from remote peer 1ca2200f67834c0fb0f43c209c09a829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.494565 8444 consensus_queue.cc:1035] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.494699 8465 consensus_queue.cc:1035] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.505149 7928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.505265 7928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.505803 7928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Sending DeleteTablet for 3 replicas of tablet 680317b7b6db4a9da44cd4bb96194c1b
I20260928 23:12:39.506017 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40312
I20260928 23:12:39.506049 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53352
I20260928 23:12:39.506112 8460 tablet_replica.cc:330] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: stopping tablet replica
I20260928 23:12:39.506161 8458 tablet_replica.cc:330] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: stopping tablet replica
I20260928 23:12:39.506174 8460 raft_consensus.cc:2229] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:39.506208 8458 raft_consensus.cc:2229] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.506248 8458 raft_consensus.cc:2258] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.506261 8460 raft_consensus.cc:2258] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.506558 8458 ts_tablet_manager.cc:1905] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.506558 8460 ts_tablet_manager.cc:1905] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.507967 8458 ts_tablet_manager.cc:1918] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:12:39.508015 8458 log.cc:1196] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/680317b7b6db4a9da44cd4bb96194c1b
I20260928 23:12:39.508082 8460 ts_tablet_manager.cc:1918] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:12:39.508152 8460 log.cc:1196] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/680317b7b6db4a9da44cd4bb96194c1b
I20260928 23:12:39.508200 8458 ts_tablet_manager.cc:1939] T 680317b7b6db4a9da44cd4bb96194c1b P 7057ec54eee14ed89258b92aa3052bbe: Deleting consensus metadata
I20260928 23:12:39.508356 8460 ts_tablet_manager.cc:1939] T 680317b7b6db4a9da44cd4bb96194c1b P 1ca2200f67834c0fb0f43c209c09a829: Deleting consensus metadata
I20260928 23:12:39.508488 7915 catalog_manager.cc:4841] TS 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679): tablet 680317b7b6db4a9da44cd4bb96194c1b (table test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]) successfully deleted
I20260928 23:12:39.508589 7912 catalog_manager.cc:4841] TS 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987): tablet 680317b7b6db4a9da44cd4bb96194c1b (table test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]) successfully deleted
I20260928 23:12:39.508750 8065 tablet_service.cc:1514] Processing DeleteTablet for tablet 680317b7b6db4a9da44cd4bb96194c1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59786
I20260928 23:12:39.508828 8459 tablet_replica.cc:330] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: stopping tablet replica
I20260928 23:12:39.508873 8459 raft_consensus.cc:2229] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.508903 8459 raft_consensus.cc:2258] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.509150 8459 ts_tablet_manager.cc:1905] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.510221 8459 ts_tablet_manager.cc:1918] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:12:39.510273 8459 log.cc:1196] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/680317b7b6db4a9da44cd4bb96194c1b
I20260928 23:12:39.510438 8459 ts_tablet_manager.cc:1939] T 680317b7b6db4a9da44cd4bb96194c1b P c96aaf1e1b724c56acdf5ea36baaac74: Deleting consensus metadata
I20260928 23:12:39.510986 7913 catalog_manager.cc:4841] TS c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089): tablet 680317b7b6db4a9da44cd4bb96194c1b (table test-table [id=4a971c9f671648e68d2a6f6b1773a0e5]) successfully deleted
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[7883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:12:39.535830 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:39.530029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.0.98.62:46609)
0928 23:12:39.530079 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.530081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.530215 (+ 134us) server_negotiation.cc:407] Connection header received
0928 23:12:39.530302 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.530303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.530312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.530359 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:39.531026 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.531519 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.532558 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.532768 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.532819 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.535761 (+ 2942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58066: BlockingRecv error: recv got EOF from 127.0.0.1:58066 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[7883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 23:12:39.543514 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.536614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.0.98.62:46609)
0928 23:12:39.536651 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.536653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.536767 (+ 114us) server_negotiation.cc:407] Connection header received
0928 23:12:39.536811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.536812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.536822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.536875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:39.537338 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.538070 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.538701 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.538970 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.539047 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.541116 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:39.541117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:39.541117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:39.541124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:39.542551 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:39.542855 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:39.542857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:39.542857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:39.542920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:39.543059 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:39.543060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:39.543060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:39.543190 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:39.543294 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.543351 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.543421 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:12:39.544914 7928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:39.545080 7928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:39.546367 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 948d0470cb5c4993a32dda675ab34001 (DEFAULT_TABLE table=test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.546396 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 948d0470cb5c4993a32dda675ab34001 (DEFAULT_TABLE table=test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.546494 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948d0470cb5c4993a32dda675ab34001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.546494 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948d0470cb5c4993a32dda675ab34001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.547775 8437 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Bootstrap starting.
I20260928 23:12:39.547837 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 948d0470cb5c4993a32dda675ab34001 (DEFAULT_TABLE table=test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.547945 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 948d0470cb5c4993a32dda675ab34001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.548185 8437 tablet_bootstrap.cc:654] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.548831 8437 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: No bootstrap required, opened a new log
I20260928 23:12:39.548884 8437 ts_tablet_manager.cc:1397] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:39.549027 8440 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Bootstrap starting.
I20260928 23:12:39.549026 8437 raft_consensus.cc:348] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.549059 8436 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Bootstrap starting.
I20260928 23:12:39.549088 8437 raft_consensus.cc:374] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.549108 8437 raft_consensus.cc:729] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.549149 8437 consensus_queue.cc:260] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.549270 8437 ts_tablet_manager.cc:1428] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:39.549400 8440 tablet_bootstrap.cc:654] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.549777 8436 tablet_bootstrap.cc:654] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.549924 8440 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: No bootstrap required, opened a new log
I20260928 23:12:39.549971 8440 ts_tablet_manager.cc:1397] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:12:39.550083 8440 raft_consensus.cc:348] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.550149 8440 raft_consensus.cc:374] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.550168 8440 raft_consensus.cc:729] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.550215 8440 consensus_queue.cc:260] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.550297 8440 ts_tablet_manager.cc:1428] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.550737 8436 tablet_bootstrap.cc:492] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: No bootstrap required, opened a new log
I20260928 23:12:39.550779 8436 ts_tablet_manager.cc:1397] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:12:39.550894 8436 raft_consensus.cc:348] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.550948 8436 raft_consensus.cc:374] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.550961 8436 raft_consensus.cc:729] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.550992 8436 consensus_queue.cc:260] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.551075 8436 ts_tablet_manager.cc:1428] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.688844 8441 raft_consensus.cc:482] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:39.688943 8441 raft_consensus.cc:504] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.689261 8441 leader_election.cc:288] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.692128 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.689454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.1:48013)
0928 23:12:39.689580 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.689589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.689650 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.689780 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.689781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.689783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.689847 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.689849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.690439 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.690439 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.691187 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.691189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.691919 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.691975 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.691979 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.692016 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.692043 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":16,"thread_start_us":57,"threads_started":1}
I20260928 23:12:39.692163 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.689502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48013 (local address 127.0.98.3:33679)
0928 23:12:39.689596 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.689598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.689637 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:12:39.689701 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.689702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.689712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.689766 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.689909 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.690364 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691270 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.691847 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691910 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.691921 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.692041 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.692084 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":40,"threads_started":1}
I20260928 23:12:39.692209 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.689508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49731 (local address 127.0.98.2:36987)
0928 23:12:39.689553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.689555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.689635 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:12:39.689692 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.689693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.689701 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.689747 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:12:39.689876 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.690334 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691329 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.691842 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691912 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.691921 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.692111 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.692152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:12:39.692306 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.689438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.98.1:49731)
0928 23:12:39.689566 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.689576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.689637 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.689754 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.689755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.689757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:12:39.689819 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.689821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.690402 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.690403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.691196 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.691199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.691966 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.691967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.692031 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.692035 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.692087 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.692123 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260928 23:12:39.692404 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948d0470cb5c4993a32dda675ab34001" candidate_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe" is_pre_election: true
I20260928 23:12:39.692468 8365 raft_consensus.cc:2454] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96aaf1e1b724c56acdf5ea36baaac74 in term 0.
I20260928 23:12:39.692631 8019 leader_election.cc:302] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7057ec54eee14ed89258b92aa3052bbe, c96aaf1e1b724c56acdf5ea36baaac74; no voters:
I20260928 23:12:39.692654 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948d0470cb5c4993a32dda675ab34001" candidate_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829" is_pre_election: true
I20260928 23:12:39.692732 8225 raft_consensus.cc:2454] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c96aaf1e1b724c56acdf5ea36baaac74 in term 0.
I20260928 23:12:39.692760 8441 raft_consensus.cc:2781] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:39.692794 8441 raft_consensus.cc:482] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:39.692813 8441 raft_consensus.cc:3037] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.693534 8441 raft_consensus.cc:504] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.693660 8441 leader_election.cc:288] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [CANDIDATE]: Term 1 election: Requested vote from peers 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987), 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679)
I20260928 23:12:39.693776 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948d0470cb5c4993a32dda675ab34001" candidate_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829"
I20260928 23:12:39.693804 8365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "948d0470cb5c4993a32dda675ab34001" candidate_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7057ec54eee14ed89258b92aa3052bbe"
I20260928 23:12:39.693845 8225 raft_consensus.cc:3037] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.693871 8365 raft_consensus.cc:3037] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.694408 8365 raft_consensus.cc:2454] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96aaf1e1b724c56acdf5ea36baaac74 in term 1.
I20260928 23:12:39.694429 8225 raft_consensus.cc:2454] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c96aaf1e1b724c56acdf5ea36baaac74 in term 1.
I20260928 23:12:39.694561 8019 leader_election.cc:302] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7057ec54eee14ed89258b92aa3052bbe, c96aaf1e1b724c56acdf5ea36baaac74; no voters:
I20260928 23:12:39.694650 8441 raft_consensus.cc:2781] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:39.694733 8441 raft_consensus.cc:686] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 LEADER]: Becoming Leader. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Running, Role: LEADER
I20260928 23:12:39.694793 8441 consensus_queue.cc:237] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.695457 7926 catalog_manager.cc:5495] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 reported cstate change: term changed from 0 to 1, leader changed from <none> to c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: UNKNOWN } } }
I20260928 23:12:39.731614 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.729111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.0.98.1:35089)
0928 23:12:39.729267 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.729270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.729278 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:39.729316 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.729317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.729342 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.729412 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.729526 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.730041 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.730819 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.731039 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.731101 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.731255 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.731382 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.731440 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.731519 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.731537 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 23:12:39.732534 8365 raft_consensus.cc:1261] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Refusing update from remote peer c96aaf1e1b724c56acdf5ea36baaac74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.732565 8225 raft_consensus.cc:1261] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Refusing update from remote peer c96aaf1e1b724c56acdf5ea36baaac74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.732811 8441 consensus_queue.cc:1035] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.732936 8507 consensus_queue.cc:1035] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.736189 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.736280 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.736871 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Sending DeleteTablet for 3 replicas of tablet 948d0470cb5c4993a32dda675ab34001
I20260928 23:12:39.737110 8065 tablet_service.cc:1514] Processing DeleteTablet for tablet 948d0470cb5c4993a32dda675ab34001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59786
I20260928 23:12:39.737102 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 948d0470cb5c4993a32dda675ab34001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40312
I20260928 23:12:39.737202 8459 tablet_replica.cc:330] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: stopping tablet replica
I20260928 23:12:39.737228 8460 tablet_replica.cc:330] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: stopping tablet replica
I20260928 23:12:39.737274 8459 raft_consensus.cc:2229] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:39.737280 8460 raft_consensus.cc:2229] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.737321 8460 raft_consensus.cc:2258] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.737349 8459 raft_consensus.cc:2258] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.737612 8460 ts_tablet_manager.cc:1905] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.737921 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 948d0470cb5c4993a32dda675ab34001 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53352
I20260928 23:12:39.737969 8459 ts_tablet_manager.cc:1905] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.737999 8458 tablet_replica.cc:330] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: stopping tablet replica
I20260928 23:12:39.738042 8458 raft_consensus.cc:2229] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.738075 8458 raft_consensus.cc:2258] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.739174 8460 ts_tablet_manager.cc:1918] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.739233 8460 log.cc:1196] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/948d0470cb5c4993a32dda675ab34001
I20260928 23:12:39.739362 8459 ts_tablet_manager.cc:1918] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.739423 8459 log.cc:1196] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/948d0470cb5c4993a32dda675ab34001
I20260928 23:12:39.739461 8460 ts_tablet_manager.cc:1939] T 948d0470cb5c4993a32dda675ab34001 P 1ca2200f67834c0fb0f43c209c09a829: Deleting consensus metadata
I20260928 23:12:39.739625 8459 ts_tablet_manager.cc:1939] T 948d0470cb5c4993a32dda675ab34001 P c96aaf1e1b724c56acdf5ea36baaac74: Deleting consensus metadata
I20260928 23:12:39.739949 8458 ts_tablet_manager.cc:1905] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.740299 7913 catalog_manager.cc:4841] TS c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089): tablet 948d0470cb5c4993a32dda675ab34001 (table test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]) successfully deleted
I20260928 23:12:39.740738 7912 catalog_manager.cc:4841] TS 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987): tablet 948d0470cb5c4993a32dda675ab34001 (table test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]) successfully deleted
I20260928 23:12:39.741362 8458 ts_tablet_manager.cc:1918] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:12:39.741420 8458 log.cc:1196] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/948d0470cb5c4993a32dda675ab34001
I20260928 23:12:39.741611 8458 ts_tablet_manager.cc:1939] T 948d0470cb5c4993a32dda675ab34001 P 7057ec54eee14ed89258b92aa3052bbe: Deleting consensus metadata
I20260928 23:12:39.741794 7926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:12:39.741961 7926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:12:39.741982 7915 catalog_manager.cc:4841] TS 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679): tablet 948d0470cb5c4993a32dda675ab34001 (table test-table [id=5f2b6367d94349a99b0ff5d8af7e191d]) successfully deleted
I20260928 23:12:39.743171 8345 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2a81963d054b43ad0e9643d410c780 (DEFAULT_TABLE table=test-table [id=612fb3c2dd724a54b40c2a0aebf0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.743165 8204 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2a81963d054b43ad0e9643d410c780 (DEFAULT_TABLE table=test-table [id=612fb3c2dd724a54b40c2a0aebf0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.743170 8065 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2a81963d054b43ad0e9643d410c780 (DEFAULT_TABLE table=test-table [id=612fb3c2dd724a54b40c2a0aebf0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:12:39.743270 8204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2a81963d054b43ad0e9643d410c780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.743269 8345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2a81963d054b43ad0e9643d410c780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.743271 8065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2a81963d054b43ad0e9643d410c780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:39.744429 8436 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Bootstrap starting.
I20260928 23:12:39.744463 8440 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Bootstrap starting.
I20260928 23:12:39.744781 8440 tablet_bootstrap.cc:654] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.744771 8436 tablet_bootstrap.cc:654] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.745026 8437 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: Bootstrap starting.
I20260928 23:12:39.745453 8436 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: No bootstrap required, opened a new log
I20260928 23:12:39.745496 8436 ts_tablet_manager.cc:1397] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:39.745527 8437 tablet_bootstrap.cc:654] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:39.745605 8436 raft_consensus.cc:348] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.745676 8436 raft_consensus.cc:374] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.745695 8436 raft_consensus.cc:729] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca2200f67834c0fb0f43c209c09a829, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.745738 8436 consensus_queue.cc:260] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.745820 8436 ts_tablet_manager.cc:1428] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.746215 8440 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: No bootstrap required, opened a new log
I20260928 23:12:39.746266 8440 ts_tablet_manager.cc:1397] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 23:12:39.746369 8440 raft_consensus.cc:348] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.746436 8440 raft_consensus.cc:374] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.746464 8440 raft_consensus.cc:729] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.746510 8440 consensus_queue.cc:260] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.746593 8440 ts_tablet_manager.cc:1428] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.747620 8437 tablet_bootstrap.cc:492] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: No bootstrap required, opened a new log
I20260928 23:12:39.747668 8437 ts_tablet_manager.cc:1397] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:12:39.747813 8437 raft_consensus.cc:348] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.747869 8437 raft_consensus.cc:374] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:39.747881 8437 raft_consensus.cc:729] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96aaf1e1b724c56acdf5ea36baaac74, State: Initialized, Role: FOLLOWER
I20260928 23:12:39.747911 8437 consensus_queue.cc:260] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.747990 8437 ts_tablet_manager.cc:1428] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:39.827992 8446 raft_consensus.cc:482] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:12:39.828088 8446 raft_consensus.cc:504] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.828256 8446 leader_election.cc:288] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987)
I20260928 23:12:39.828449 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2a81963d054b43ad0e9643d410c780" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" is_pre_election: true
I20260928 23:12:39.828449 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2a81963d054b43ad0e9643d410c780" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829" is_pre_election: true
I20260928 23:12:39.828542 8225 raft_consensus.cc:2454] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 0.
I20260928 23:12:39.828542 8085 raft_consensus.cc:2454] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 0.
I20260928 23:12:39.828703 8296 leader_election.cc:302] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ca2200f67834c0fb0f43c209c09a829, 7057ec54eee14ed89258b92aa3052bbe; no voters:
I20260928 23:12:39.828816 8446 raft_consensus.cc:2781] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:12:39.828845 8446 raft_consensus.cc:482] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:12:39.828863 8446 raft_consensus.cc:3037] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.829627 8446 raft_consensus.cc:504] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.829746 8446 leader_election.cc:288] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 election: Requested vote from peers c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089), 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987)
I20260928 23:12:39.829903 8225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2a81963d054b43ad0e9643d410c780" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ca2200f67834c0fb0f43c209c09a829"
I20260928 23:12:39.829945 8085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2a81963d054b43ad0e9643d410c780" candidate_uuid: "7057ec54eee14ed89258b92aa3052bbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c96aaf1e1b724c56acdf5ea36baaac74"
I20260928 23:12:39.829993 8225 raft_consensus.cc:3037] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.830014 8085 raft_consensus.cc:3037] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:39.830572 8085 raft_consensus.cc:2454] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 1.
I20260928 23:12:39.830705 8297 leader_election.cc:302] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7057ec54eee14ed89258b92aa3052bbe, c96aaf1e1b724c56acdf5ea36baaac74; no voters:
I20260928 23:12:39.830793 8446 raft_consensus.cc:2781] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:39.830775 8225 raft_consensus.cc:2454] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7057ec54eee14ed89258b92aa3052bbe in term 1.
I20260928 23:12:39.830884 8446 raft_consensus.cc:686] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 1 LEADER]: Becoming Leader. State: Replica: 7057ec54eee14ed89258b92aa3052bbe, State: Running, Role: LEADER
I20260928 23:12:39.830935 8446 consensus_queue.cc:237] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } }
I20260928 23:12:39.831454 7926 catalog_manager.cc:5495] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7057ec54eee14ed89258b92aa3052bbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7057ec54eee14ed89258b92aa3052bbe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33679 } health_report { overall_health: HEALTHY } } }
I20260928 23:12:39.845534 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.843162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.0.98.3:33679)
0928 23:12:39.843196 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.843198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.843207 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:39.843260 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.843261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.843270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.843307 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.843421 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.843883 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.844891 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.845072 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.845112 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.845295 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.845334 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.845386 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.845454 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.845476 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:12:39.849953 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.847674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36987 (local address 127.0.98.3:39401)
0928 23:12:39.847809 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.847820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.847875 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.847989 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.847990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.847991 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:12:39.848045 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.848047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.848613 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.848614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.849189 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.849191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.849559 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.849560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.849623 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.849635 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.849804 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.849806 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.849845 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.849885 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 23:12:39.849938 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.847865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39401 (local address 127.0.98.2:36987)
0928 23:12:39.847909 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.847910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.847916 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:39.847936 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.847937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.847945 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.847988 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.848101 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.848550 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.849303 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.849504 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.849557 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.849698 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.849757 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.849798 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.849847 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.849865 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:12:39.851790 8085 raft_consensus.cc:1261] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Refusing update from remote peer 7057ec54eee14ed89258b92aa3052bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.851836 8225 raft_consensus.cc:1261] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Refusing update from remote peer 7057ec54eee14ed89258b92aa3052bbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:12:39.852029 8519 consensus_queue.cc:1035] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "c96aaf1e1b724c56acdf5ea36baaac74" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.852185 8519 consensus_queue.cc:1035] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ca2200f67834c0fb0f43c209c09a829" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:12:39.858738 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.856644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56087 (local address 127.0.98.3:33679)
0928 23:12:39.856680 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:39.856682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:39.856688 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:39.856736 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:39.856737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:39.856745 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:39.856787 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:12:39.856874 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.857417 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.858111 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.858306 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.858359 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.858522 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.858547 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.858587 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:39.858638 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 23:12:39.858659 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:12:39.858748 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.856598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33679 (local address 127.0.98.2:56087)
0928 23:12:39.856643 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.856654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:39.856688 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:39.856787 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:39.856788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:39.856789 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 23:12:39.856832 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.856833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.857501 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.857502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.858043 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:39.858045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:39.858360 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:39.858361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:39.858413 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:39.858419 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 23:12:39.858594 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 23:12:39.858596 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:39.858636 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:12:39.858686 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:12:39.864866 7926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.864964 7926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:12:39.865535 7926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60bd2b13a0ec42359e35f6e5f25b9288: Sending DeleteTablet for 3 replicas of tablet 2d2a81963d054b43ad0e9643d410c780
I20260928 23:12:39.865756 8204 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2a81963d054b43ad0e9643d410c780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40312
I20260928 23:12:39.865785 8065 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2a81963d054b43ad0e9643d410c780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59786
I20260928 23:12:39.865816 8345 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2a81963d054b43ad0e9643d410c780 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:12:39 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53352
I20260928 23:12:39.865859 8460 tablet_replica.cc:330] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: stopping tablet replica
I20260928 23:12:39.865878 8459 tablet_replica.cc:330] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74: stopping tablet replica
I20260928 23:12:39.865899 8460 raft_consensus.cc:2229] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.865916 8459 raft_consensus.cc:2229] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:12:39.865929 8460 raft_consensus.cc:2258] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.865944 8459 raft_consensus.cc:2258] T 2d2a81963d054b43ad0e9643d410c780 P c96aaf1e1b724c56acdf5ea36baaac74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.865983 8458 tablet_replica.cc:330] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: stopping tablet replica
I20260928 23:12:39.866035 8458 raft_consensus.cc:2229] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:12:39.866096 8458 raft_consensus.cc:2258] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:12:39.866349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8000
I20260928 23:12:39.866362 8458 ts_tablet_manager.cc:1905] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.866374 8460 ts_tablet_manager.cc:1905] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:12:39.867957 8460 ts_tablet_manager.cc:1918] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:12:39.867957 8458 ts_tablet_manager.cc:1918] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 23:12:39.868016 8460 log.cc:1196] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/2d2a81963d054b43ad0e9643d410c780
I20260928 23:12:39.868016 8458 log.cc:1196] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/2d2a81963d054b43ad0e9643d410c780
I20260928 23:12:39.868212 8458 ts_tablet_manager.cc:1939] T 2d2a81963d054b43ad0e9643d410c780 P 7057ec54eee14ed89258b92aa3052bbe: Deleting consensus metadata
I20260928 23:12:39.868209 8460 ts_tablet_manager.cc:1939] T 2d2a81963d054b43ad0e9643d410c780 P 1ca2200f67834c0fb0f43c209c09a829: Deleting consensus metadata
I20260928 23:12:39.868546 7912 catalog_manager.cc:4841] TS 1ca2200f67834c0fb0f43c209c09a829 (127.0.98.2:36987): tablet 2d2a81963d054b43ad0e9643d410c780 (table test-table [id=612fb3c2dd724a54b40c2a0aebf0c1d8]) successfully deleted
I20260928 23:12:39.868628 7915 catalog_manager.cc:4841] TS 7057ec54eee14ed89258b92aa3052bbe (127.0.98.3:33679): tablet 2d2a81963d054b43ad0e9643d410c780 (table test-table [id=612fb3c2dd724a54b40c2a0aebf0c1d8]) successfully deleted
W20260928 23:12:39.873181 7913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35089 (error 108)
I20260928 23:12:39.873446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8140
I20260928 23:12:39.874058 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:39.873546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35089 (local address 127.0.0.1:59804)
0928 23:12:39.873989 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:39.874023 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":36,"threads_started":1}
W20260928 23:12:39.874183 7913 catalog_manager.cc:4584] TS c96aaf1e1b724c56acdf5ea36baaac74 (127.0.98.1:35089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d2a81963d054b43ad0e9643d410c780: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35089: connect: Connection refused (error 111)
I20260928 23:12:39.879654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8280
I20260928 23:12:39.884443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 7896
2026-09-28T23:12:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2615 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[8531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[8531](info): set up 2 sockets
Sep 28 23:12:39 dist-test-slave-6vgd krb5kdc[8531](info): commencing operation
krb5kdc: starting...
W20260928 23:12:40.833422 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:12:40 dist-test-slave-6vgd krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:12:40.942330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34985
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:41.020185 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:41.020383 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:41.020403 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:41.021498 8544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:41.021530 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:41.021544 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:41.021555 8544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:41.021572 8544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:41.022994 8544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:41.023176 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:41.023339 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:41.027720 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:41.027710 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:41.027731 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:41.027818 8544 server_base.cc:1034] running on GCE node
I20260928 23:12:41.028151 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:41.028414 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:41.029544 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790637161029529 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T23:12:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1145 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:41 dist-test-slave-6vgd krb5kdc[8557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:41 dist-test-slave-6vgd krb5kdc[8557](info): set up 2 sockets
Sep 28 23:12:41 dist-test-slave-6vgd krb5kdc[8557](info): commencing operation
krb5kdc: starting...
W20260928 23:12:41.973414 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:12:42.081347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:42.157620 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:42.157807 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:42.157826 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:42.158946 8570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:42.158983 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:42.158996 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:42.159009 8570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:42.159031 8570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:42.160494 8570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:42.160712 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:42.160902 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:42.165344 8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.165356 8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.165411 8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.165428 8570 server_base.cc:1034] running on GCE node
I20260928 23:12:42.165743 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:42.165982 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:42.167145 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790637162167129 us; error 28 us; skew 500 ppm
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:42.169349 8570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:12:42.169723 8570 webserver.cc:460] Webserver started at http://127.0.98.62:36525/ using document root <none> and password file <none>
I20260928 23:12:42.169910 8570 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:42.169955 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:42.170086 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:42.171062 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "aaa920d5feead16103dff4eb16b4166c"
server_key_iv: "68768339b6627c19721d8d6551e39854"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.171372 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "aaa920d5feead16103dff4eb16b4166c"
server_key_iv: "68768339b6627c19721d8d6551e39854"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.172552 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:42.173146 8585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.173312 8570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:42.173393 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "aaa920d5feead16103dff4eb16b4166c"
server_key_iv: "68768339b6627c19721d8d6551e39854"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.173451 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:42.189950 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:42.191725 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:42.191852 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:42.195438 8570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44651
I20260928 23:12:42.195504 8637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44651 every 8 connection(s)
I20260928 23:12:42.195914 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:42.196552 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:42.198853 8638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Bootstrap starting.
I20260928 23:12:42.199391 8638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:42.199657 8638 log.cc:824] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:42.200307 8638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: No bootstrap required, opened a new log
I20260928 23:12:42.201859 8638 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.201987 8638 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:42.202008 8638 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2f1d678acb4c5c98ad97ee98c1eb1e, State: Initialized, Role: FOLLOWER
I20260928 23:12:42.202147 8638 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.202209 8638 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:42.202230 8638 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:42.202275 8638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:42.203061 8638 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.203155 8638 leader_election.cc:302] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2f1d678acb4c5c98ad97ee98c1eb1e; no voters:
I20260928 23:12:42.203299 8638 leader_election.cc:288] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:42.203341 8641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:42.203487 8641 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 LEADER]: Becoming Leader. State: Replica: cc2f1d678acb4c5c98ad97ee98c1eb1e, State: Running, Role: LEADER
I20260928 23:12:42.203521 8638 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:42.203568 8641 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.203908 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2f1d678acb4c5c98ad97ee98c1eb1e. Latest consensus state: current_term: 1 leader_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260928 23:12:42.203904 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260928 23:12:42.204070 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:42.204263 8650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:42.204422 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:42.204747 8650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:42.205330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8570
I20260928 23:12:42.205432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:42.205605 392 external_mini_cluster.cc:1428] Setting key 80830affd4c0fb4b29f5dec13c9e3c46
I20260928 23:12:42.206184 8650 catalog_manager.cc:1347] Generated new cluster ID: b35adbb57c204118b16e818a5b22b9cd
I20260928 23:12:42.206228 8650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:42.214794 8650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:42.215243 8650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:42.217051 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.206337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.0.98.62:44651)
0928 23:12:42.206464 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:42.206472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:42.206500 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:12:42.206601 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:42.206607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:42.206656 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:42.206771 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:42.207509 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.208846 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.209911 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.210958 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.211056 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.213687 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:42.213690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:42.213691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:42.213702 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:42.215601 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.215944 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.215947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.215947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.216052 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.216216 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.216217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.216217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.216478 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:42.216602 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:42.216690 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:12:42.216832 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260928 23:12:42.220508 8650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Generated new TSK 0
I20260928 23:12:42.220662 8650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:12:42.247516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:42.325177 8664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:42.325394 8664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:42.325415 8664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:42.326574 8664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:42.326617 8664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:42.326642 8664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:42.328090 8664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:42.328299 8664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:42.328505 8664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:42.333289 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.333271 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.333258 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.333271 8664 server_base.cc:1034] running on GCE node
I20260928 23:12:42.333681 8664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:42.333894 8664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:42.335031 8664 hybrid_clock.cc:648] HybridClock initialized: now 1790637162335019 us; error 28 us; skew 500 ppm
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:42.341324 8664 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:12:42.341737 8664 webserver.cc:460] Webserver started at http://127.0.98.1:44331/ using document root <none> and password file <none>
I20260928 23:12:42.341944 8664 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:42.341992 8664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:42.342128 8664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:42.343144 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b53ad1a277a483db12ab6fce05de74e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "c3c9cb3e9beeff181af59f7ced9184ed"
server_key_iv: "ad80ae480ba027034a89272d8315a646"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.343473 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b53ad1a277a483db12ab6fce05de74e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "c3c9cb3e9beeff181af59f7ced9184ed"
server_key_iv: "ad80ae480ba027034a89272d8315a646"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.344666 8664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:42.345329 8679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.345477 8664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.345561 8664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "9b53ad1a277a483db12ab6fce05de74e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "c3c9cb3e9beeff181af59f7ced9184ed"
server_key_iv: "ad80ae480ba027034a89272d8315a646"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.345626 8664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:42.354373 8664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:42.356328 8664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:42.356453 8664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:42.356647 8664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:42.356907 8664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:42.356938 8664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.356966 8664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:42.356988 8664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.362296 8664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33797
I20260928 23:12:42.362354 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33797 every 8 connection(s)
I20260928 23:12:42.362617 8664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:42.371232 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.363286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33771 (local address 127.0.98.62:44651)
0928 23:12:42.363347 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:42.363349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:42.363526 (+ 177us) server_negotiation.cc:407] Connection header received
0928 23:12:42.364428 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:42.364429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:42.364444 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:42.364494 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:42.365087 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.365636 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.366591 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.366800 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.366853 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.368809 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:42.368811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:42.368811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:42.368817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:42.370092 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.370414 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.370414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.370415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.370457 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.370654 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.370656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.370656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.370796 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:42.370859 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:42.371098 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 23:12:42.371167 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:12:42.371333 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.363253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:33771)
0928 23:12:42.363418 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:42.363453 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:42.364330 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:42.364573 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:42.364578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:42.364817 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:42.365000 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.365004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.365715 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.365716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.366507 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.366509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.366865 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.366866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.366988 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.367423 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:42.367431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:42.368728 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:42.370154 (+ 1426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.370159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.370161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.370359 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.370499 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.370499 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.370499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.370584 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.370853 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:42.370855 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:42.370964 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:12:42.371037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:12:42.371162 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260928 23:12:42.371517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8664
I20260928 23:12:42.371608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:42.371812 392 external_mini_cluster.cc:1428] Setting key e9e3e114b1c4d53230dfb556c7bbaec7
I20260928 23:12:42.371800 8793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260928 23:12:42.371889 8793 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:42.372061 8793 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260928 23:12:42.372541 8602 ts_manager.cc:194] Registered new tserver with Master: 9b53ad1a277a483db12ab6fce05de74e (127.0.98.1:33797)
I20260928 23:12:42.373365 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33771
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:12:42.392942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:42.471239 8798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:42.471438 8798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:42.471460 8798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:42.472555 8798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:42.472590 8798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:42.472615 8798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:42.474114 8798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:42.474308 8798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:42.474488 8798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:42.479333 8806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.479311 8803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.479372 8798 server_base.cc:1034] running on GCE node
W20260928 23:12:42.479315 8804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.479691 8798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:42.479924 8798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:42.481089 8798 hybrid_clock.cc:648] HybridClock initialized: now 1790637162481072 us; error 32 us; skew 500 ppm
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:42.483575 8798 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:12:42.484025 8798 webserver.cc:460] Webserver started at http://127.0.98.2:34699/ using document root <none> and password file <none>
I20260928 23:12:42.484241 8798 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:42.484289 8798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:42.484413 8798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:42.485466 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ac7412048f747d4b3e05a6302bc6ce7"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "8202c8958f66a17f7d4409154762093d"
server_key_iv: "9d62da1f63f22969fae32adeff74306c"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.485872 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ac7412048f747d4b3e05a6302bc6ce7"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "8202c8958f66a17f7d4409154762093d"
server_key_iv: "9d62da1f63f22969fae32adeff74306c"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.487079 8798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:42.487757 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.487965 8798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:42.488050 8798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "0ac7412048f747d4b3e05a6302bc6ce7"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "8202c8958f66a17f7d4409154762093d"
server_key_iv: "9d62da1f63f22969fae32adeff74306c"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.488113 8798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:42.518672 8798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:42.520857 8798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:42.520993 8798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:42.521241 8798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:42.521556 8798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:42.521589 8798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.521617 8798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:42.521641 8798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.526993 8798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35031
I20260928 23:12:42.527060 8926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35031 every 8 connection(s)
I20260928 23:12:42.527346 8798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:42.535620 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.528042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39237 (local address 127.0.98.62:44651)
0928 23:12:42.528102 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:42.528104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:42.528278 (+ 174us) server_negotiation.cc:407] Connection header received
0928 23:12:42.529155 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:42.529155 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:42.529165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:42.529255 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:42.529858 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.530344 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.531281 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.531454 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.531496 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.533378 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:42.533379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:42.533379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:42.533386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:42.534635 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.534948 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.534948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.534949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.534993 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.535159 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.535160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.535160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.535253 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:42.535295 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:42.535491 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:12:42.535555 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:12:42.535796 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.528006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:39237)
0928 23:12:42.528159 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:42.528202 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:42.529074 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:42.529312 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:42.529315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:42.529557 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:42.529780 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.529787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.530412 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.530412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.531221 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.531222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.531505 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.531505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.531597 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.532019 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:42.532026 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:42.533307 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:42.534695 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.534698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.534700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.534897 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.535035 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.535035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.535036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.535119 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.535293 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:42.535296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:42.535406 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 23:12:42.535480 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:12:42.535593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260928 23:12:42.536201 8927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260928 23:12:42.536283 8927 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:42.536425 8927 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260928 23:12:42.536758 8602 ts_manager.cc:194] Registered new tserver with Master: 0ac7412048f747d4b3e05a6302bc6ce7 (127.0.98.2:35031)
I20260928 23:12:42.537106 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39237
I20260928 23:12:42.537391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8798
I20260928 23:12:42.537487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:42.537662 392 external_mini_cluster.cc:1428] Setting key a828e2bfa54c8b55576e233f6d482317
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:12:42.557883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:42.636646 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:42.636857 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:42.636884 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:42.638096 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:42.638140 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:42.638166 8932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:42.639647 8932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:42.639845 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:42.640027 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:42.644853 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.644886 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.644925 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.644985 8932 server_base.cc:1034] running on GCE node
I20260928 23:12:42.645349 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:42.645604 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:42.646749 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790637162646738 us; error 30 us; skew 500 ppm
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:42.649293 8932 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:12:42.649757 8932 webserver.cc:460] Webserver started at http://127.0.98.3:34361/ using document root <none> and password file <none>
I20260928 23:12:42.649960 8932 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:42.650007 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:42.650131 8932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:42.651158 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "52e32f90d4fb435a9f81cbaeddb43b77"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "1035883d15d8f1ecde1cc97d406ffe1b"
server_key_iv: "fc148a8ffe4380c29d157b8772c2afc5"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.651487 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "52e32f90d4fb435a9f81cbaeddb43b77"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "1035883d15d8f1ecde1cc97d406ffe1b"
server_key_iv: "fc148a8ffe4380c29d157b8772c2afc5"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.652725 8932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:42.653427 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.653570 8932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:42.653647 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "52e32f90d4fb435a9f81cbaeddb43b77"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "1035883d15d8f1ecde1cc97d406ffe1b"
server_key_iv: "fc148a8ffe4380c29d157b8772c2afc5"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.653712 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:42.677055 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:42.678930 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:42.679068 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:42.679281 8932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:42.679579 8932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:42.679611 8932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.679641 8932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:42.679661 8932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.685089 8932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40811
I20260928 23:12:42.685129 9060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40811 every 8 connection(s)
I20260928 23:12:42.685438 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:12:42.692448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 8932
I20260928 23:12:42.692554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:42.692719 392 external_mini_cluster.cc:1428] Setting key 3a1fa2173ff2dbc6f436e3576a45d431
I20260928 23:12:42.694352 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.686134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60075 (local address 127.0.98.62:44651)
0928 23:12:42.686209 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:42.686211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:42.686357 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:12:42.687248 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:42.687249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:42.687259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:42.687294 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:42.687857 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.688343 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.689305 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.689492 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.689544 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.691600 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:42.691601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:42.691602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:42.691608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:42.693050 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.693505 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.693506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.693506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.693570 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:42.693805 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:42.693805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:42.693806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:42.693907 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:42.693953 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:42.694186 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 23:12:42.694248 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:12:42.694465 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:42.686098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:60075)
0928 23:12:42.686247 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:42.686281 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:42.687154 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:42.687356 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:42.687359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:42.687613 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:42.687800 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.687803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.688404 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.688404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.689204 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:42.689233 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:42.689602 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:42.689603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:42.689735 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:42.690157 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:42.690166 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:42.691516 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:42.693128 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.693136 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.693138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.693440 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.693632 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:12:42.693632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:12:42.693633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:12:42.693763 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:12:42.693950 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:42.693952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:42.694088 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 23:12:42.694168 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 23:12:42.694285 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260928 23:12:42.694854 9061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260928 23:12:42.694948 9061 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:42.695119 9061 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260928 23:12:42.695484 8602 ts_manager.cc:194] Registered new tserver with Master: 52e32f90d4fb435a9f81cbaeddb43b77 (127.0.98.3:40811)
I20260928 23:12:42.695849 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60075
I20260928 23:12:42.704574 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 23:12:42.773828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8664
I20260928 23:12:42.777639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8798
I20260928 23:12:42.781414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8932
I20260928 23:12:42.784871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 8570
I20260928 23:12:42.789145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--webserver_interface=127.0.98.62
--webserver_port=36525
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:12:42.866685 9070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:42.866917 9070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:42.866945 9070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:42.868101 9070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:42.868140 9070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:42.868158 9070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:42.868175 9070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:42.868188 9070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:42.869741 9070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:42.869988 9070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:42.870216 9070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:42.874946 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.875083 9070 server_base.cc:1034] running on GCE node
W20260928 23:12:42.874933 9078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:42.874933 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:42.875407 9070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:42.875691 9070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:42.876850 9070 hybrid_clock.cc:648] HybridClock initialized: now 1790637162876833 us; error 30 us; skew 500 ppm
Sep 28 23:12:42 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:42.879326 9070 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:12:42.879750 9070 webserver.cc:460] Webserver started at http://127.0.98.62:36525/ using document root <none> and password file <none>
I20260928 23:12:42.879948 9070 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:42.880003 9070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:42.881378 9070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:42.882010 9085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.882155 9070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:42.882233 9070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "aaa920d5feead16103dff4eb16b4166c"
server_key_iv: "68768339b6627c19721d8d6551e39854"
server_key_version: "encryptionkey@0"
I20260928 23:12:42.882578 9070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:42.898245 9070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:42.900092 9070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:42.900218 9070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:42.904024 9070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44651
I20260928 23:12:42.904084 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44651 every 8 connection(s)
I20260928 23:12:42.904539 9070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:42.905807 9138 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:12:42.906390 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Bootstrap starting.
I20260928 23:12:42.913482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9070
I20260928 23:12:42.913825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33797
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44651
--webserver_port=44331
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:12:42.914095 9138 log.cc:824] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:42.915824 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:12:42.916059 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Bootstrap complete.
I20260928 23:12:42.918155 9138 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.918367 9138 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2f1d678acb4c5c98ad97ee98c1eb1e, State: Initialized, Role: FOLLOWER
I20260928 23:12:42.918509 9138 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.918567 9138 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:42.918608 9138 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:42.918653 9138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:12:42.919593 9138 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.919679 9138 leader_election.cc:302] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2f1d678acb4c5c98ad97ee98c1eb1e; no voters:
I20260928 23:12:42.919835 9138 leader_election.cc:288] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:12:42.919885 9142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:12:42.920030 9142 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [term 2 LEADER]: Becoming Leader. State: Replica: cc2f1d678acb4c5c98ad97ee98c1eb1e, State: Running, Role: LEADER
I20260928 23:12:42.920084 9138 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:42.920166 9142 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260928 23:12:42.920506 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2f1d678acb4c5c98ad97ee98c1eb1e. Latest consensus state: current_term: 2 leader_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260928 23:12:42.920575 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:42.920840 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2f1d678acb4c5c98ad97ee98c1eb1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260928 23:12:42.920913 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:42.921028 9148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:42.921541 9148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:42.921731 9148 catalog_manager.cc:1259] Loaded cluster ID: b35adbb57c204118b16e818a5b22b9cd
I20260928 23:12:42.921871 9148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:42.922655 9148 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:42.922772 9148 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc2f1d678acb4c5c98ad97ee98c1eb1e: Loaded TSK: 0
I20260928 23:12:42.922904 9148 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:12:43.008383 9141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:43.008579 9141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:43.008610 9141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:43.009768 9141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:43.009809 9141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:43.009845 9141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:43.011227 9141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44331
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:43.011438 9141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:43.011631 9141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:43.016279 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:43.016259 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:43.016702 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:43.016918 9141 server_base.cc:1034] running on GCE node
I20260928 23:12:43.017115 9141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:43.017356 9141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:43.018474 9141 hybrid_clock.cc:648] HybridClock initialized: now 1790637163018460 us; error 33 us; skew 500 ppm
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:43.020964 9141 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 23:12:43.021404 9141 webserver.cc:460] Webserver started at http://127.0.98.1:44331/ using document root <none> and password file <none>
I20260928 23:12:43.021612 9141 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:43.021663 9141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:43.022960 9141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:43.023511 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.023660 9141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.023738 9141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "9b53ad1a277a483db12ab6fce05de74e"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "c3c9cb3e9beeff181af59f7ced9184ed"
server_key_iv: "ad80ae480ba027034a89272d8315a646"
server_key_version: "encryptionkey@0"
I20260928 23:12:43.024042 9141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:43.041044 9141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:43.043130 9141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:43.043264 9141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:43.043470 9141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:43.043746 9141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:43.043778 9141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.043807 9141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:43.043829 9141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.049180 9141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33797
I20260928 23:12:43.049268 9287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33797 every 8 connection(s)
I20260928 23:12:43.049597 9141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.058405 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.050275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:32883)
0928 23:12:43.050473 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.050518 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.051106 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.051702 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.051706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.052230 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.052407 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.052412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.053962 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.053963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.054730 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.054732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.055492 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.055493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.055615 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.056113 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.056124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.058269 (+ 2145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
W20260928 23:12:43.058578 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.050381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32883 (local address 127.0.98.62:44651)
0928 23:12:43.051304 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.051313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.051347 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:12:43.051475 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.051481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.051533 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.051661 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.052530 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.053872 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.054804 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.055395 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.055490 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.058422 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32883: BlockingRecv error: recv got EOF from 127.0.98.1:32883 (error 108)
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":64,"threads_started":1}
I20260928 23:12:43.058736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9141
W20260928 23:12:43.058871 9288 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44651 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44651: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 23:12:43.059019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35031
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44651
--webserver_port=34699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.063320 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.059091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:49967)
0928 23:12:43.059142 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.059152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.059183 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.059322 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.059323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.059452 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.059500 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.059501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.060244 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.060245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.060978 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.060980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.061360 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.061361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.061422 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.061798 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.061801 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.063276 (+ 1475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 23:12:43.063441 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.059130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49967 (local address 127.0.98.62:44651)
0928 23:12:43.059161 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.059162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.059173 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:12:43.059244 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.059245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.059255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.059561 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.059587 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.060167 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.061044 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.061296 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.061347 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.063386 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49967: BlockingRecv error: recv got EOF from 127.0.98.1:49967 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.067528 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.063543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:55367)
0928 23:12:43.063617 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.063626 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.063658 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.063799 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.063799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.063891 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.063915 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.063916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.064696 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.064696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.065157 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.065158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.065573 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.065573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.065608 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.065979 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.065983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.067484 (+ 1501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 23:12:43.067634 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.063638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55367 (local address 127.0.98.62:44651)
0928 23:12:43.063688 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.063689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.063698 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:43.063725 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.063725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.063736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.063796 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.063970 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.064629 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.065244 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.065511 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.065578 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.067579 (+ 2001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55367: BlockingRecv error: recv got EOF from 127.0.98.1:55367 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 23:12:43.067628 9288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:12:43.137914 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:43.138126 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:43.138155 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:43.139277 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:43.139317 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:43.139351 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:43.140749 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34699
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:43.140978 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:43.141180 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:43.145753 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:43.145715 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:43.145892 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:43.145893 9292 server_base.cc:1034] running on GCE node
I20260928 23:12:43.146112 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:43.146325 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:43.147468 9292 hybrid_clock.cc:648] HybridClock initialized: now 1790637163147450 us; error 30 us; skew 500 ppm
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:43.149842 9292 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 23:12:43.150244 9292 webserver.cc:460] Webserver started at http://127.0.98.2:34699/ using document root <none> and password file <none>
I20260928 23:12:43.150444 9292 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:43.150506 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:43.151737 9292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:12:43.152230 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.152374 9292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.152448 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "0ac7412048f747d4b3e05a6302bc6ce7"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "8202c8958f66a17f7d4409154762093d"
server_key_iv: "9d62da1f63f22969fae32adeff74306c"
server_key_version: "encryptionkey@0"
I20260928 23:12:43.152769 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:43.164181 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:43.166172 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:43.166302 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:43.166488 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:43.166765 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:43.166796 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.166823 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:43.166847 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.172196 9292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35031
I20260928 23:12:43.172271 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35031 every 8 connection(s)
I20260928 23:12:43.172555 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:43.173732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9292
I20260928 23:12:43.174016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40811
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44651
--webserver_port=34361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:12:43.180285 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.173190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39673 (local address 127.0.98.62:44651)
0928 23:12:43.173264 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.173266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.173458 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:12:43.174128 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.174129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.174145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.174206 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.175339 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.176054 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.177011 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.177331 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.177404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.180239 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39673: BlockingRecv error: recv got EOF from 127.0.98.2:39673 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:12:43.180392 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.173173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:39673)
0928 23:12:43.173350 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.173384 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.174012 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.174276 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.174281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.174968 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.175238 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.175245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.176138 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.176139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.176925 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.176926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.177405 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.177406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.177510 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.177995 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.178002 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.180171 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260928 23:12:43.180598 9421 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44651 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44651: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.185016 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.180799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:34849)
0928 23:12:43.180907 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.180917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.180971 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.181078 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.181079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.181246 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.181297 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.181299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.181883 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.181884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.182663 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.182666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.182954 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.182955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.183017 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.183353 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.183358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.184929 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 23:12:43.185034 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.180859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34849 (local address 127.0.98.62:44651)
0928 23:12:43.180906 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.180907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.180957 (+ 50us) server_negotiation.cc:407] Connection header received
0928 23:12:43.181024 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.181025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.181033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.181072 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.181365 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.181815 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.182736 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.182899 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.182936 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.184992 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34849: BlockingRecv error: recv got EOF from 127.0.98.2:34849 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.189237 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.185255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:42577)
0928 23:12:43.185295 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.185303 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.185334 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.185446 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.185446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.185567 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.185591 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.185592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.186455 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.186455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.186921 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.186922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.187326 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.187327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.187363 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.187693 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.187697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.189172 (+ 1475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:12:43.189335 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.185305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42577 (local address 127.0.98.62:44651)
0928 23:12:43.185343 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.185345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.185353 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:43.185374 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.185375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.185384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.185443 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.185656 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.186381 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.186980 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.187272 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.187328 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.189276 (+ 1948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42577: BlockingRecv error: recv got EOF from 127.0.98.2:42577 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:12:43.189400 9421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:12:43.254199 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:43.254385 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:43.254406 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:43.255738 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:43.255792 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:43.255818 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:43.257325 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34361
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:43.257511 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:43.257689 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:43.262346 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:43.262490 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:43.262521 9424 server_base.cc:1034] running on GCE node
W20260928 23:12:43.262591 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:43.262784 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:43.263020 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:43.264160 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790637163264149 us; error 28 us; skew 500 ppm
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637163, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:12:43.266407 9424 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 23:12:43.266809 9424 webserver.cc:460] Webserver started at http://127.0.98.3:34361/ using document root <none> and password file <none>
I20260928 23:12:43.266991 9424 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:43.267040 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:43.268234 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:12:43.268694 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.268833 9424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.268910 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "52e32f90d4fb435a9f81cbaeddb43b77"
format_stamp: "Formatted at 2026-09-28 23:12:42 on dist-test-slave-6vgd"
server_key: "1035883d15d8f1ecde1cc97d406ffe1b"
server_key_iv: "fc148a8ffe4380c29d157b8772c2afc5"
server_key_version: "encryptionkey@0"
I20260928 23:12:43.269245 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:43.281592 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:43.283195 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:43.283324 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:43.283510 9424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:43.283771 9424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:43.283803 9424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.283830 9424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:43.283854 9424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:43.289284 9424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40811
I20260928 23:12:43.289320 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40811 every 8 connection(s)
I20260928 23:12:43.289630 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 23:12:43.295953 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.290255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60723 (local address 127.0.98.62:44651)
0928 23:12:43.290332 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.290335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.290494 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:12:43.290950 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.290951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.290960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.291005 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.291852 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.292322 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.293251 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.293420 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.293462 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.295907 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60723: BlockingRecv error: recv got EOF from 127.0.98.3:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:12:43.295969 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.290243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:60723)
0928 23:12:43.290385 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.290418 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.290876 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.291064 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.291067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.291595 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.291788 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.291792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.292383 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.292384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.293171 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.293173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.293470 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.293471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.293567 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.294015 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.294023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.295834 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
W20260928 23:12:43.296165 9553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44651 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:44651: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 23:12:43.298516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9424
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.300217 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.296359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:39359)
0928 23:12:43.296397 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.296407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.296440 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.296656 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.296657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.296751 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.296789 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.296790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.297360 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.297361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.297834 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.297836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.298092 (+ 256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.298093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.298134 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.298492 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.298496 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.300153 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:12:43.300259 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.296516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39359 (local address 127.0.98.62:44651)
0928 23:12:43.296558 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.296559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.296566 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:43.296584 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.296585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.296592 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.296630 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.296839 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.297312 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.297885 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.298044 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.298093 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.300220 (+ 2127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39359: BlockingRecv error: recv got EOF from 127.0.98.3:39359 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:43 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:43.304105 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.300478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:45769)
0928 23:12:43.300572 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:43.300582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:43.300621 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:43.300730 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:43.300730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:43.300844 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:43.300867 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.300868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.301557 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.301558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.302009 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:43.302010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.302280 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.302281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:43.302333 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.302643 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:43.302646 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:43.304054 (+ 1408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 23:12:43.304169 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:43.300533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45769 (local address 127.0.98.62:44651)
0928 23:12:43.300577 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.300579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.300609 (+ 30us) server_negotiation.cc:407] Connection header received
0928 23:12:43.300671 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.300672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.300679 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.300722 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.300926 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.301494 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.302065 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.302219 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.302278 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.304119 (+ 1841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45769: BlockingRecv error: recv got EOF from 127.0.98.3:45769 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 23:12:43.304212 9553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:12:43.306020 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:43.298870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.98.62:44651)
0928 23:12:43.298972 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:43.298974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:43.299046 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:12:43.299117 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:43.299117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:43.299125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:43.299172 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:43.299799 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.300426 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.301196 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:43.301497 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:43.301559 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:43.303195 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:43.303200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:43.303200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:12:43.303209 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:43.304821 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:43.305110 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:43.305114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:43.305114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:43.305227 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:12:43.305382 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:12:43.305382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:12:43.305383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:12:43.305607 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:43.305734 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:43.305833 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:12:43.305933 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:44.072216 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:44.067998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:60993)
0928 23:12:44.068147 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:44.068159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:44.068215 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:44.068364 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:44.068366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:44.068532 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:44.068578 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.068580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.069181 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.069182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.069751 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.069753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.070062 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.070063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.070107 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.070560 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:44.070565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:44.072146 (+ 1581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
W20260928 23:12:44.072295 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:44.068099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60993 (local address 127.0.98.62:44651)
0928 23:12:44.068191 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:44.068192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:44.068199 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:44.068268 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:44.068269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:44.068280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:44.068345 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:44.068631 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.069124 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.069816 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.070001 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.070043 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.072229 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60993: BlockingRecv error: recv got EOF from 127.0.98.1:60993 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":48,"threads_started":1}
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:44.193791 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:44.189733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:43311)
0928 23:12:44.189842 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:44.189852 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:44.189904 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:44.190016 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:44.190017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:44.190168 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:44.190214 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.190216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.190797 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.190798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.191319 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.191321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.191587 (+ 266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.191587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.191621 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.192061 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:44.192065 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:44.193726 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
W20260928 23:12:44.193867 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:44.189790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43311 (local address 127.0.98.62:44651)
0928 23:12:44.189846 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:44.189848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:44.189893 (+ 45us) server_negotiation.cc:407] Connection header received
0928 23:12:44.189955 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:44.189956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:44.189966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:44.190002 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:44.190264 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.190741 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.191373 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.191542 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.191583 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.193814 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43311: BlockingRecv error: recv got EOF from 127.0.98.2:43311 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:44 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:44.308663 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:44.304555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:49979)
0928 23:12:44.304688 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:44.304699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:44.304769 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:44.304892 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:44.304893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:44.305057 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:44.305101 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.305102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.305754 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.305755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.306300 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:44.306303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.306619 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.306620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:44.306672 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.307076 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:44.307080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:44.308577 (+ 1497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
W20260928 23:12:44.308724 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:44.304614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49979 (local address 127.0.98.62:44651)
0928 23:12:44.304671 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:44.304673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:44.304768 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:12:44.304819 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:44.304820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:44.304834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:44.304880 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:44.305167 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.305686 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.306383 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:44.306562 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:44.306615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:44.308673 (+ 2058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49979: BlockingRecv error: recv got EOF from 127.0.98.3:49979 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:45.076843 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:45.072627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:35429)
0928 23:12:45.072760 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:45.072772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:45.072833 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:45.072959 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:45.072960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:45.073117 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:45.073164 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.073165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.073953 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.073954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.074482 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.074484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.074779 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.074780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.074814 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.075204 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:45.075208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:45.076785 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
W20260928 23:12:45.076949 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:45.072693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35429 (local address 127.0.98.62:44651)
0928 23:12:45.072812 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:45.072814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:45.072821 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:45.072886 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:45.072887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:45.072898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:45.072947 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:45.073228 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.073899 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.074543 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.074730 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.074776 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.076875 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35429: BlockingRecv error: recv got EOF from 127.0.98.1:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:45.198357 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:45.194235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:42517)
0928 23:12:45.194366 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:45.194378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:45.194437 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:45.194556 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:45.194557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:45.194707 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:45.194754 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.194756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.195338 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.195339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.195863 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.195865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.196172 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.196173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.196207 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.196612 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:45.196616 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:45.198285 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
W20260928 23:12:45.198503 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:45.194308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42517 (local address 127.0.98.62:44651)
0928 23:12:45.194361 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:45.194363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:45.194431 (+ 68us) server_negotiation.cc:407] Connection header received
0928 23:12:45.194478 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:45.194479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:45.194488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:45.194546 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:45.194800 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.195281 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.195933 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.196119 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.196171 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.198438 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42517: BlockingRecv error: recv got EOF from 127.0.98.2:42517 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:45 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:45.313194 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:45.309086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:55185)
0928 23:12:45.309243 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:45.309256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:45.309318 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:45.309439 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:45.309440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:45.309624 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:45.309669 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.309671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.310291 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.310293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.310817 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:45.310819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.311144 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.311145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:45.311197 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.311611 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:45.311615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:45.313136 (+ 1521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
W20260928 23:12:45.313304 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:45.309189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55185 (local address 127.0.98.62:44651)
0928 23:12:45.309257 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:45.309259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:45.309301 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:12:45.309365 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:45.309365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:45.309378 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:45.309421 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:45.309741 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.310224 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.310889 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:45.311083 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:45.311140 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:45.313244 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55185: BlockingRecv error: recv got EOF from 127.0.98.3:55185 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:46.081847 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:46.077289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:56413)
0928 23:12:46.077441 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:46.077454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:46.077524 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:46.077743 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:46.077745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:46.077966 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:46.078024 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.078027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.078700 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.078701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.079257 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.079259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.079565 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.079566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.079606 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.080015 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:46.080019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:46.081782 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
W20260928 23:12:46.081940 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:46.077497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56413 (local address 127.0.98.62:44651)
0928 23:12:46.077617 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:46.077619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:46.077625 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:12:46.077645 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:46.077645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:46.077657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:46.077723 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:46.078090 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.078613 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.079322 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.079510 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.079556 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.081864 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56413: BlockingRecv error: recv got EOF from 127.0.98.1:56413 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:46.202849 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:46.198840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:52515)
0928 23:12:46.199006 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:46.199017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:46.199080 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:46.199211 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:46.199212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:46.199359 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:46.199407 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.199409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.199990 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.199990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.200497 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.200498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.200778 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.200778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.200817 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.201237 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:46.201241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:46.202784 (+ 1543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
W20260928 23:12:46.202927 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:46.198920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52515 (local address 127.0.98.62:44651)
0928 23:12:46.199000 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:46.199002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:46.199072 (+ 70us) server_negotiation.cc:407] Connection header received
0928 23:12:46.199138 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:46.199139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:46.199150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:46.199196 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:46.199455 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.199937 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.200548 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.200722 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.200770 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.202869 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52515: BlockingRecv error: recv got EOF from 127.0.98.2:52515 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:46 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:46.317947 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:46.313629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:55305)
0928 23:12:46.313764 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:46.313776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:46.313840 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:46.313964 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:46.313965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:46.314141 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:46.314186 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.314188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.314827 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.314829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.315348 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:46.315350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.315678 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.315679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:46.315731 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.316129 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:46.316133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:46.317878 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260928 23:12:46.318066 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:46.313688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55305 (local address 127.0.98.62:44651)
0928 23:12:46.313738 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:46.313740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:46.313832 (+ 92us) server_negotiation.cc:407] Connection header received
0928 23:12:46.313891 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:46.313892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:46.313905 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:46.313950 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:46.314255 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.314751 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.315423 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:46.315619 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:46.315673 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:46.318004 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55305: BlockingRecv error: recv got EOF from 127.0.98.3:55305 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:47.086617 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:47.082336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:44015)
0928 23:12:47.082457 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:47.082467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:47.082519 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:47.082637 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:47.082638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:47.082793 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:47.082839 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.082840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.083437 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.083437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.083967 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.083969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.084411 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.084412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.084470 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.084906 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:47.084910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:47.086550 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
W20260928 23:12:47.086694 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:47.082397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44015 (local address 127.0.98.62:44651)
0928 23:12:47.082517 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:47.082519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:47.082529 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:12:47.082555 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:47.082556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:47.082566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:47.082628 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:47.082891 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.083385 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.084039 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.084313 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.084376 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.086632 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44015: BlockingRecv error: recv got EOF from 127.0.98.1:44015 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:47.207185 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:47.203277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:45417)
0928 23:12:47.203397 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:47.203409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:47.203466 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:47.203579 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:47.203580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:47.203726 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:47.203772 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.203773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.204348 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.204348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.204859 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.204860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.205125 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.205126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.205160 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.205574 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:47.205578 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:47.207126 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
W20260928 23:12:47.207276 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:47.203327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45417 (local address 127.0.98.62:44651)
0928 23:12:47.203413 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:47.203415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:47.203461 (+ 46us) server_negotiation.cc:407] Connection header received
0928 23:12:47.203519 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:47.203519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:47.203531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:47.203568 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:47.203818 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.204299 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.204910 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.205081 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.205122 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.207218 (+ 2096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45417: BlockingRecv error: recv got EOF from 127.0.98.2:45417 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:47 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:47.323410 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:47.318528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:38289)
0928 23:12:47.318700 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:47.318713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:47.318784 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:47.318925 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:47.318926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:47.319159 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:47.319211 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.319214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.319956 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.319957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.320489 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:47.320491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.320882 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.320882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:47.320936 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.321373 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:47.321378 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:47.323328 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
W20260928 23:12:47.323679 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:47.318675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38289 (local address 127.0.98.62:44651)
0928 23:12:47.318722 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:47.318724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:47.318764 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:12:47.318836 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:47.318837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:47.318849 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:47.318908 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:47.319310 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.319881 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.320566 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:47.320798 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:47.320878 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:47.323448 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38289: BlockingRecv error: recv got EOF from 127.0.98.3:38289 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:48.091204 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:48.087077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:44887)
0928 23:12:48.087188 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:48.087198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:48.087246 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:48.087380 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:48.087381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:48.087537 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:48.087585 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.087586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.088180 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.088181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.088715 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.088717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.088997 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.088998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.089034 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.089459 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:48.089463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:48.091130 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
W20260928 23:12:48.091296 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:48.087136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44887 (local address 127.0.98.62:44651)
0928 23:12:48.087262 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:48.087263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:48.087272 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:48.087293 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:48.087293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:48.087303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:48.087375 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:48.087635 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.088123 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.088771 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.088951 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.088994 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.091237 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44887: BlockingRecv error: recv got EOF from 127.0.98.1:44887 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:48.211985 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:48.207649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:33713)
0928 23:12:48.207809 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:48.207820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:48.207872 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:48.207995 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:48.207996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:48.208141 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:48.208187 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.208189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.208791 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.208791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.209379 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.209382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.209838 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.209839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.209882 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.210291 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:48.210308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:48.211909 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
W20260928 23:12:48.212042 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:48.207788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33713 (local address 127.0.98.62:44651)
0928 23:12:48.207843 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:48.207845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:48.207856 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:12:48.207921 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:48.207922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:48.207932 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:48.207984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:48.208241 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.208736 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.209454 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.209776 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.209829 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.211994 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33713: BlockingRecv error: recv got EOF from 127.0.98.2:33713 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:48 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:48.328842 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:48.323931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:56777)
0928 23:12:48.324080 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:48.324091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:48.324160 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:48.324284 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:48.324285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:48.324458 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:48.324503 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.324505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.325135 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.325136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.325718 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:48.325721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.326161 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.326162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:48.326208 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.326700 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:48.326706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:48.328761 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":69,"threads_started":1}
W20260928 23:12:48.328924 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:48.324007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56777 (local address 127.0.98.62:44651)
0928 23:12:48.324070 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:48.324072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:48.324140 (+ 68us) server_negotiation.cc:407] Connection header received
0928 23:12:48.324210 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:48.324211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:48.324225 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:48.324270 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:48.324572 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.325063 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.325816 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:48.326081 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:48.326149 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:48.328863 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56777: BlockingRecv error: recv got EOF from 127.0.98.3:56777 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:49.095705 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:49.091625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:58031)
0928 23:12:49.091742 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:49.091754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:49.091817 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:49.091948 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:49.091948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:49.092102 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:49.092146 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.092147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.092745 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.092746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.093303 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.093305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.093590 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.093591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.093627 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.094016 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:49.094020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:49.095636 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
W20260928 23:12:49.095791 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:49.091687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58031 (local address 127.0.98.62:44651)
0928 23:12:49.091812 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:49.091814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:49.091822 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:49.091874 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:49.091874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:49.091884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:49.091937 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:49.092193 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.092683 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.093358 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.093540 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.093589 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.095721 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58031: BlockingRecv error: recv got EOF from 127.0.98.1:58031 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:49.216838 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:49.212451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:57185)
0928 23:12:49.212615 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:49.212625 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:49.212697 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:49.212829 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:49.212830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:49.213026 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:49.213070 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.213072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.213761 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.213762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.214295 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.214297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.214625 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.214626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.214677 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.215096 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:49.215101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:49.216771 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
W20260928 23:12:49.216915 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:49.212535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57185 (local address 127.0.98.62:44651)
0928 23:12:49.212595 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:49.212597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:49.212690 (+ 93us) server_negotiation.cc:407] Connection header received
0928 23:12:49.212752 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:49.212753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:49.212765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:49.212812 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:49.213146 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.213684 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.214378 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.214565 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.214620 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.216855 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57185: BlockingRecv error: recv got EOF from 127.0.98.2:57185 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:49 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:49.334538 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:49.329282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:60939)
0928 23:12:49.329443 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:49.329460 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:49.329535 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:49.329680 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:49.329682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:49.329932 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:49.329988 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.329990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.330856 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.330857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.331623 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:49.331626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.332082 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.332083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:49.332147 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.332610 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:49.332615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:49.334465 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260928 23:12:49.334632 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:49.329493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60939 (local address 127.0.98.62:44651)
0928 23:12:49.329571 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:49.329573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:49.329580 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:12:49.329604 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:49.329605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:49.329618 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:49.329666 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:49.330064 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.330775 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.331708 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:49.331999 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:49.332079 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:49.334568 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60939: BlockingRecv error: recv got EOF from 127.0.98.3:60939 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:50.100569 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:50.096160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:57735)
0928 23:12:50.096300 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:50.096311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:50.096362 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:50.096500 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:50.096500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:50.096662 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:50.096709 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.096711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.097397 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.097398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.097934 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.097936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.098302 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.098303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.098358 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.098875 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:50.098880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:50.100510 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
W20260928 23:12:50.100677 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:50.096260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57735 (local address 127.0.98.62:44651)
0928 23:12:50.096367 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:50.096368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:50.096377 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:50.096403 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:50.096403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:50.096413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:50.096477 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:50.096780 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.097321 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.098017 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.098228 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.098272 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.100599 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57735: BlockingRecv error: recv got EOF from 127.0.98.1:57735 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:50.221565 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:50.217342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:37943)
0928 23:12:50.217449 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:50.217460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:50.217515 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:50.217636 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:50.217637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:50.217788 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:50.217836 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.217837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.218431 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.218431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.218965 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.218967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.219281 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.219282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.219319 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.219768 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:50.219772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:50.221504 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260928 23:12:50.221683 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:50.217375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37943 (local address 127.0.98.62:44651)
0928 23:12:50.217420 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:50.217466 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:50.217501 (+ 35us) server_negotiation.cc:407] Connection header received
0928 23:12:50.217566 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:50.217567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:50.217578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:50.217632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:50.217887 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.218377 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.219049 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.219232 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.219277 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.221593 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37943: BlockingRecv error: recv got EOF from 127.0.98.2:37943 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:50 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:50.339426 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:50.335001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:36727)
0928 23:12:50.335191 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:50.335203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:50.335273 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:50.335425 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:50.335426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:50.335614 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:50.335672 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.335675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.336482 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.336483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.336994 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:50.336996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.337317 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.337318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:50.337356 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.337770 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:50.337775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:50.339367 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":62,"threads_started":1}
W20260928 23:12:50.339496 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:50.335066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36727 (local address 127.0.98.62:44651)
0928 23:12:50.335108 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:50.335111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:50.335256 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:12:50.335344 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:50.335345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:50.335360 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:50.335411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:50.335748 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.336412 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.337054 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:50.337259 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:50.337316 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:50.339435 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36727: BlockingRecv error: recv got EOF from 127.0.98.3:36727 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:51.105098 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:51.101018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:43951)
0928 23:12:51.101152 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:51.101163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:51.101230 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:51.101368 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:51.101369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:51.101525 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:51.101569 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.101571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.102184 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.102185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.102700 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.102701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.102985 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.102986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.103021 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.103449 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:51.103453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:51.105041 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
W20260928 23:12:51.105227 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:51.101138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43951 (local address 127.0.98.62:44651)
0928 23:12:51.101260 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:51.101262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:51.101270 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:51.101289 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:51.101290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:51.101300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:51.101357 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:51.101626 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.102126 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.102754 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.102939 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.102984 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.105137 (+ 2153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43951: BlockingRecv error: recv got EOF from 127.0.98.1:43951 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":49,"threads_started":1}
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:51.226254 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:51.221994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:55611)
0928 23:12:51.222108 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:51.222119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:51.222177 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:51.222287 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:51.222287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:51.222436 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:51.222483 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.222484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.223235 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.223235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.223756 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.223757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.224055 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.224056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.224099 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.224511 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:51.224516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:51.226196 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
W20260928 23:12:51.226337 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:51.222058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55611 (local address 127.0.98.62:44651)
0928 23:12:51.222109 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:51.222110 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:51.222163 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:12:51.222218 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:51.222219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:51.222229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:51.222276 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:51.222530 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.223179 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.223820 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.224005 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.224050 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.226280 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55611: BlockingRecv error: recv got EOF from 127.0.98.2:55611 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:51 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:51.344173 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:51.339888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:58519)
0928 23:12:51.340009 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:51.340020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:51.340084 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:51.340244 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:51.340245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:51.340469 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:51.340532 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.340534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.341243 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.341245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.341784 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:51.341786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.342072 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.342073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:51.342108 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.342493 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:51.342497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:51.344105 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260928 23:12:51.344247 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:51.339917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58519 (local address 127.0.98.62:44651)
0928 23:12:51.339958 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:51.339960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:51.340077 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:12:51.340148 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:51.340149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:51.340162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:51.340218 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:51.340608 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.341136 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.341843 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:51.342020 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:51.342066 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:51.344199 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58519: BlockingRecv error: recv got EOF from 127.0.98.3:58519 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:52.109980 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:52.105582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:42993)
0928 23:12:52.105719 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:52.105730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:52.105789 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:52.105949 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:52.105950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:52.106115 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:52.106165 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.106166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.106842 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.106843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.107399 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.107401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.107688 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.107689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.107725 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.108223 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:52.108227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:52.109922 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
W20260928 23:12:52.110097 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:52.105684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42993 (local address 127.0.98.62:44651)
0928 23:12:52.105803 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:52.105805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:52.105813 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:52.105834 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:52.105835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:52.105845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:52.105906 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:52.106274 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.106765 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.107459 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.107639 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.107680 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.110017 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42993: BlockingRecv error: recv got EOF from 127.0.98.1:42993 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:52.231070 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:52.226689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:54279)
0928 23:12:52.226820 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:52.226831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:52.226886 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:52.227007 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:52.227008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:52.227157 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:52.227204 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.227205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.227863 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.227864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.228400 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.228402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.228763 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.228764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.228805 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.229289 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:52.229293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:52.231011 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20260928 23:12:52.231172 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:52.226757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54279 (local address 127.0.98.62:44651)
0928 23:12:52.226810 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:52.226812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:52.226882 (+ 70us) server_negotiation.cc:407] Connection header received
0928 23:12:52.226929 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:52.226929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:52.226939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:52.226991 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:52.227255 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.227777 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.228497 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.228697 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.228756 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.231114 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54279: BlockingRecv error: recv got EOF from 127.0.98.2:54279 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:52 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:52.348985 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:52.344616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:39795)
0928 23:12:52.344729 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:52.344740 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:52.344795 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:52.344928 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:52.344928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:52.345076 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:52.345123 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.345124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.345780 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.345781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.346313 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:52.346315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.346625 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.346626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:52.346660 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.347053 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:52.347057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:52.348901 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
W20260928 23:12:52.349042 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:52.344711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39795 (local address 127.0.98.62:44651)
0928 23:12:52.344776 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:52.344778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:52.344788 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:12:52.344843 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:52.344844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:52.344854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:52.344918 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:52.345171 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.345706 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.346388 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:52.346570 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:52.346623 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:52.348976 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39795: BlockingRecv error: recv got EOF from 127.0.98.3:39795 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:53.114698 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:53.110482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:35421)
0928 23:12:53.110624 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:53.110635 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:53.110702 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:53.110835 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:53.110836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:53.111006 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:53.111050 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.111052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.111687 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.111688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.112206 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.112208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.112528 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.112529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.112580 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.113013 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:53.113018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:53.114622 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
W20260928 23:12:53.114773 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:53.110493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35421 (local address 127.0.98.62:44651)
0928 23:12:53.110609 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:53.110611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:53.110697 (+ 86us) server_negotiation.cc:407] Connection header received
0928 23:12:53.110754 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:53.110755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:53.110767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:53.110823 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:53.111119 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.111618 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.112280 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.112471 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.112524 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.114712 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35421: BlockingRecv error: recv got EOF from 127.0.98.1:35421 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:53.235766 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:53.231499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:35497)
0928 23:12:53.231657 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:53.231668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:53.231738 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:53.231872 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:53.231873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:53.232052 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:53.232098 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.232100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.232747 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.232748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.233323 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.233325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.233639 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.233640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.233690 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.234111 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:53.234115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:53.235688 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
W20260928 23:12:53.235843 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:53.231581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35497 (local address 127.0.98.62:44651)
0928 23:12:53.231642 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:53.231645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:53.231725 (+ 80us) server_negotiation.cc:407] Connection header received
0928 23:12:53.231795 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:53.231796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:53.231809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:53.231866 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:53.232166 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.232675 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.233398 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.233585 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.233635 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.235783 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35497: BlockingRecv error: recv got EOF from 127.0.98.2:35497 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:53 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:53.353489 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:53.349413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:35375)
0928 23:12:53.349588 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:53.349602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:53.349670 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:53.349779 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:53.349779 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:53.349932 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:53.349977 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.349978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.350582 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.350583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.351101 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:53.351103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.351372 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.351372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:53.351410 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.351785 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:53.351788 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:53.353412 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
W20260928 23:12:53.353556 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:53.349558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35375 (local address 127.0.98.62:44651)
0928 23:12:53.349608 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:53.349610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:53.349649 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:12:53.349724 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:53.349724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:53.349736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:53.349771 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:53.350032 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.350525 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.351153 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:53.351325 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:53.351367 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:53.353502 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35375: BlockingRecv error: recv got EOF from 127.0.98.3:35375 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:54.119681 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:54.115188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:47647)
0928 23:12:54.115320 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:54.115330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:54.115378 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:54.115542 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:54.115543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:54.115700 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:54.115753 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.115754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.116375 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.116376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.116930 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.116932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.117332 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.117333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.117385 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.117815 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:54.117819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:54.119606 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260928 23:12:54.119782 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:54.115248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47647 (local address 127.0.98.62:44651)
0928 23:12:54.115355 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:54.115357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:54.115366 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:54.115456 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:54.115457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:54.115467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:54.115530 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:54.115816 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.116311 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.117023 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.117241 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.117311 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.119699 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47647: BlockingRecv error: recv got EOF from 127.0.98.1:47647 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:54.240278 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:54.236209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:53145)
0928 23:12:54.236350 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:54.236359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:54.236412 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:54.236533 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:54.236534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:54.236690 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:54.236737 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.236739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.237351 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.237351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.237867 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.237868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.238158 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.238158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.238194 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.238586 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:54.238590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:54.240220 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
W20260928 23:12:54.240375 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:54.236291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53145 (local address 127.0.98.62:44651)
0928 23:12:54.236344 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:54.236346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:54.236407 (+ 61us) server_negotiation.cc:407] Connection header received
0928 23:12:54.236467 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:54.236468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:54.236478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:54.236517 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:54.236784 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.237301 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.237923 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.238112 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.238154 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.240314 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53145: BlockingRecv error: recv got EOF from 127.0.98.2:53145 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:54 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:54.358075 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:54.353908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:51619)
0928 23:12:54.354026 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:54.354037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:54.354093 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:54.354208 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:54.354209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:54.354355 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:54.354400 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.354401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.355046 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.355047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.355621 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:54.355623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.355950 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.355950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:54.355987 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.356408 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:54.356412 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:54.358017 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
W20260928 23:12:54.358160 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:54.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51619 (local address 127.0.98.62:44651)
0928 23:12:54.354025 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:54.354027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:54.354089 (+ 62us) server_negotiation.cc:407] Connection header received
0928 23:12:54.354131 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:54.354131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:54.354144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:54.354201 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:54.354452 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.354989 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.355702 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:54.355888 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:54.355947 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:54.358101 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51619: BlockingRecv error: recv got EOF from 127.0.98.3:51619 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:55.124611 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:55.120195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:51961)
0928 23:12:55.120326 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:55.120337 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:55.120395 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:55.120515 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:55.120516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:55.120669 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:55.120714 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.120716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.121518 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.121519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.122079 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.122081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.122403 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.122404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.122444 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.122846 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:55.122850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:55.124538 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260928 23:12:55.124697 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:55.120246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51961 (local address 127.0.98.62:44651)
0928 23:12:55.120351 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:55.120353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:55.120392 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:12:55.120436 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:55.120437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:55.120447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:55.120507 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:55.120775 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.121458 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.122146 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.122343 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.122387 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.124628 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51961: BlockingRecv error: recv got EOF from 127.0.98.1:51961 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:55.244989 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:55.240713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:33473)
0928 23:12:55.240840 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:55.240851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:55.240920 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:55.241050 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:55.241051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:55.241261 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:55.241315 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.241317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.241937 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.241939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.242477 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.242479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.242804 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.242805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.242857 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.243303 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:55.243308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:55.244908 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260928 23:12:55.245047 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:55.240792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33473 (local address 127.0.98.62:44651)
0928 23:12:55.240843 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:55.240845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:55.240905 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:12:55.240978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:55.240979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:55.240992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:55.241037 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:55.241381 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.241869 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.242552 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.242742 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.242799 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.244997 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33473: BlockingRecv error: recv got EOF from 127.0.98.2:33473 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:55 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:55.362649 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:55.358483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:56555)
0928 23:12:55.358620 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:55.358632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:55.358707 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:55.358828 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:55.358830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:55.358991 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:55.359034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.359036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.359665 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.359666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.360202 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:55.360204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.360506 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.360507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:55.360562 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.360944 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:55.360948 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:55.362555 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260928 23:12:55.362684 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:55.358562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56555 (local address 127.0.98.62:44651)
0928 23:12:55.358635 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:55.358637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:55.358688 (+ 51us) server_negotiation.cc:407] Connection header received
0928 23:12:55.358756 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:55.358757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:55.358770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:55.358817 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:55.359095 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.359599 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.360271 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:55.360451 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:55.360502 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:55.362633 (+ 2131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56555: BlockingRecv error: recv got EOF from 127.0.98.3:56555 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:56.129732 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:56.125148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:55025)
0928 23:12:56.125332 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:56.125344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:56.125413 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:56.125537 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:56.125538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:56.125692 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:56.125739 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.125741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.126595 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.126596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.127165 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.127168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.127498 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.127499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.127538 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.127989 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:56.127993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:56.129673 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
W20260928 23:12:56.129820 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:56.125260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55025 (local address 127.0.98.62:44651)
0928 23:12:56.125356 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:56.125358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:56.125409 (+ 51us) server_negotiation.cc:407] Connection header received
0928 23:12:56.125458 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:56.125458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:56.125469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:56.125527 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:56.125812 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.126533 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.127254 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.127445 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.127489 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.129764 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55025: BlockingRecv error: recv got EOF from 127.0.98.1:55025 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:56.249660 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:56.245423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:41559)
0928 23:12:56.245571 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:56.245582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:56.245628 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:56.245760 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:56.245761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:56.245911 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:56.245958 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.245960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.246607 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.246608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.247140 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.247141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.247467 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.247467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.247507 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.247918 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:56.247922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:56.249590 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260928 23:12:56.249724 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:56.245537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41559 (local address 127.0.98.62:44651)
0928 23:12:56.245608 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:56.245610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:56.245618 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:56.245685 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:56.245686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:56.245696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:56.245746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:56.246015 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.246527 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.247218 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.247407 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.247460 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.249671 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41559: BlockingRecv error: recv got EOF from 127.0.98.2:41559 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:56 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:56.367007 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:56.363041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:44281)
0928 23:12:56.363156 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:56.363167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:56.363220 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:56.363321 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:56.363321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:56.363465 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:56.363510 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.363511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.364078 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.364078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.364579 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:56.364581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.364858 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.364859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:56.364895 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.365369 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:56.365373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:56.366948 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260928 23:12:56.367096 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:56.363070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44281 (local address 127.0.98.62:44651)
0928 23:12:56.363116 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:56.363118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:56.363214 (+ 96us) server_negotiation.cc:407] Connection header received
0928 23:12:56.363259 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:56.363260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:56.363272 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:56.363315 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:56.363551 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.364028 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.364631 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:56.364803 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:56.364847 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:56.367037 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44281: BlockingRecv error: recv got EOF from 127.0.98.3:44281 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:57.134442 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:57.130230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:34387)
0928 23:12:57.130344 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:57.130354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:57.130407 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:57.130534 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:57.130534 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:57.130683 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:57.130730 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.130731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.131350 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.131351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.131878 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.131880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.132164 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.132165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.132199 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.132588 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:57.132592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:57.134366 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260928 23:12:57.134533 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:57.130315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34387 (local address 127.0.98.62:44651)
0928 23:12:57.130431 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:57.130432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:57.130440 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:12:57.130459 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:57.130460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:57.130469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:57.130524 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:57.130783 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.131283 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.131940 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.132119 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.132162 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.134470 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34387: BlockingRecv error: recv got EOF from 127.0.98.1:34387 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":44,"threads_started":1}
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:57.254519 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:57.250068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:35201)
0928 23:12:57.250169 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:57.250196 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:57.250253 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:57.250400 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:57.250400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:57.250587 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:57.250643 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.250645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.251305 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.251306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.251980 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.251983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.252436 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.252437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.252498 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.252923 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:57.252928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:57.254462 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
W20260928 23:12:57.254596 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:57.250103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35201 (local address 127.0.98.62:44651)
0928 23:12:57.250165 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:57.250167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:57.250240 (+ 73us) server_negotiation.cc:407] Connection header received
0928 23:12:57.250311 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:57.250312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:57.250325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:57.250390 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:57.250724 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.251237 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.252078 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.252361 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.252418 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.254542 (+ 2124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35201: BlockingRecv error: recv got EOF from 127.0.98.2:35201 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:57 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:57.371361 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:57.367400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:41181)
0928 23:12:57.367534 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:57.367543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:57.367600 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:57.367696 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:57.367697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:57.367842 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:57.367888 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.367889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.368475 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.368476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.369002 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:57.369004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.369296 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.369297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:57.369337 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.369709 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:57.369713 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:57.371302 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260928 23:12:57.371440 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:57.367454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41181 (local address 127.0.98.62:44651)
0928 23:12:57.367498 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:57.367500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:57.367585 (+ 85us) server_negotiation.cc:407] Connection header received
0928 23:12:57.367638 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:57.367639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:57.367649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:57.367692 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:57.367930 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.368403 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.369056 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:57.369247 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:57.369297 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:57.371381 (+ 2084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41181: BlockingRecv error: recv got EOF from 127.0.98.3:41181 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 23:12:58 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:58 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:58.143636 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.134888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:60181)
0928 23:12:58.135007 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:58.135017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:58.135080 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:58.135197 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:58.135198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:58.135351 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:58.135397 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.135399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.136012 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.136013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.136539 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.136541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.136830 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.136831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.136866 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.141906 (+ 5040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:58.141912 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:58.143564 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260928 23:12:58.143793 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:58.134968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60181 (local address 127.0.98.62:44651)
0928 23:12:58.135070 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.135073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.135082 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:12:58.135120 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.135120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.135130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.135185 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.135448 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.135941 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.136601 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.136782 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.136827 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.143721 (+ 6894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60181: BlockingRecv error: recv got EOF from 127.0.98.1:60181 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
Sep 28 23:12:58 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 23:12:58 dist-test-slave-6vgd krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 23:12:58.259220 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.255028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:55407)
0928 23:12:58.255163 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:58.255174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:58.255228 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:58.255343 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:58.255343 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:58.255511 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:58.255557 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.255559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.256234 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.256236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.256774 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.256776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.257108 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.257109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.257148 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.257586 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:12:58.257590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:58.259161 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:44651: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
W20260928 23:12:58.259320 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:58.255084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55407 (local address 127.0.98.62:44651)
0928 23:12:58.255128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.255130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.255213 (+ 83us) server_negotiation.cc:407] Connection header received
0928 23:12:58.255271 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.255272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.255281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.255330 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.255612 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.256133 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.256852 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.257049 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.257102 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.259251 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55407: BlockingRecv error: recv got EOF from 127.0.98.2:55407 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:12:58.305367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9141
I20260928 23:12:58.309273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9292
I20260928 23:12:58.312776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9424
I20260928 23:12:58.316187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9070
2026-09-28T23:12:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17285 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T23:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:58Z Disabled control of system clock
I20260928 23:12:58.341102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41857
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 23:12:58.419317 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:58.419508 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:58.419526 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:58.420634 9620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:58.420670 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:58.420682 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:58.420693 9620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:58.420706 9620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:58.422097 9620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:58.422295 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:58.422530 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:58.427357 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.427476 9620 server_base.cc:1034] running on GCE node
W20260928 23:12:58.427393 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:58.427393 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.427824 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:58.428107 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:58.429261 9620 hybrid_clock.cc:648] HybridClock initialized: now 1790637178429247 us; error 30 us; skew 500 ppm
I20260928 23:12:58.430450 9620 webserver.cc:460] Webserver started at http://127.0.98.62:43511/ using document root <none> and password file <none>
I20260928 23:12:58.430681 9620 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:58.430729 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:58.430848 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:58.431846 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "7c549e5a8dbb4f95806fc2869e0450ff"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "932349b8389e2443c38741a4a6038f66"
server_key_iv: "d84eb2b9b6c4475c59d70d4e6e9f28c1"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.432178 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c549e5a8dbb4f95806fc2869e0450ff"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "932349b8389e2443c38741a4a6038f66"
server_key_iv: "d84eb2b9b6c4475c59d70d4e6e9f28c1"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.433347 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:12:58.434015 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.434181 9620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.434262 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "7c549e5a8dbb4f95806fc2869e0450ff"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "932349b8389e2443c38741a4a6038f66"
server_key_iv: "d84eb2b9b6c4475c59d70d4e6e9f28c1"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.434329 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:58.445832 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:58.446153 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:58.446285 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:58.449887 9620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41857
I20260928 23:12:58.449922 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41857 every 8 connection(s)
I20260928 23:12:58.450227 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:58.450814 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:58.453315 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff: Bootstrap starting.
I20260928 23:12:58.453855 9687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:58.454099 9687 log.cc:824] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:58.454571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9620
I20260928 23:12:58.454664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:58.454777 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff: No bootstrap required, opened a new log
I20260928 23:12:58.454864 392 external_mini_cluster.cc:1428] Setting key b909639212b40e69e9ad6b8e8c29a54c
I20260928 23:12:58.456471 9687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } }
I20260928 23:12:58.456614 9687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:58.456646 9687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c549e5a8dbb4f95806fc2869e0450ff, State: Initialized, Role: FOLLOWER
I20260928 23:12:58.456789 9687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } }
I20260928 23:12:58.456873 9687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:58.456902 9687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:58.456948 9687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:58.457636 9687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } }
I20260928 23:12:58.457742 9687 leader_election.cc:302] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c549e5a8dbb4f95806fc2869e0450ff; no voters:
I20260928 23:12:58.457882 9687 leader_election.cc:288] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:58.457911 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:58.458089 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [term 1 LEADER]: Becoming Leader. State: Replica: 7c549e5a8dbb4f95806fc2869e0450ff, State: Running, Role: LEADER
I20260928 23:12:58.458106 9687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:58.458254 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } }
I20260928 23:12:58.458654 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c549e5a8dbb4f95806fc2869e0450ff. Latest consensus state: current_term: 1 leader_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } } }
I20260928 23:12:58.458734 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:58.459240 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:58.458937 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c549e5a8dbb4f95806fc2869e0450ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41857 } } }
I20260928 23:12:58.459743 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:58.459916 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:58.461563 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.455407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.62:41857)
0928 23:12:58.455612 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.455621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.455646 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:12:58.456444 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.456450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.456503 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.456626 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.456845 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.458537 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.459570 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.460167 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.460282 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.460903 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:58.460907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:58.460907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:58.461070 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:58.461107 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:58.461121 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:58.461182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:58.461317 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 23:12:58.461398 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260928 23:12:58.462045 9700 catalog_manager.cc:1347] Generated new cluster ID: 5e864601843c463f84f3b27404ed6a3b
I20260928 23:12:58.462090 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:58.472738 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:58.473197 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:58.483794 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c549e5a8dbb4f95806fc2869e0450ff: Generated new TSK 0
I20260928 23:12:58.483958 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:12:58.490770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--builtin_ntp_servers=127.0.98.20:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:12:58.582165 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:58.582357 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:58.582376 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:58.583477 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:58.583514 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:58.583537 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:58.584792 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:58.584996 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:58.585327 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:58.590749 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.591012 9711 server_base.cc:1034] running on GCE node
W20260928 23:12:58.590734 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:58.590719 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.591307 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:58.591540 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:58.592691 9711 hybrid_clock.cc:648] HybridClock initialized: now 1790637178592671 us; error 35 us; skew 500 ppm
I20260928 23:12:58.593842 9711 webserver.cc:460] Webserver started at http://127.0.98.1:44055/ using document root <none> and password file <none>
I20260928 23:12:58.594038 9711 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:58.594081 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:58.594197 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:58.595206 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "2938b7c26b784480a23c0ec9570241ad"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "408b23562a5094cf3ac3dd20b81a8334"
server_key_iv: "3b2efef87cbee133dbfd570720a52d7e"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.595541 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2938b7c26b784480a23c0ec9570241ad"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "408b23562a5094cf3ac3dd20b81a8334"
server_key_iv: "3b2efef87cbee133dbfd570720a52d7e"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.596676 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:58.597316 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.597446 9711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:58.597520 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "2938b7c26b784480a23c0ec9570241ad"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "408b23562a5094cf3ac3dd20b81a8334"
server_key_iv: "3b2efef87cbee133dbfd570720a52d7e"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.597585 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:58.611655 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:58.611904 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:58.612020 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:58.612202 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:58.612478 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:58.612510 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.612535 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:58.612560 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.617781 9711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40341
I20260928 23:12:58.617817 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40341 every 8 connection(s)
I20260928 23:12:58.618093 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:58.622659 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.618725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58605 (local address 127.0.98.62:41857)
0928 23:12:58.618765 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.618766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.618989 (+ 223us) server_negotiation.cc:407] Connection header received
0928 23:12:58.619429 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.619430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.619440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.619485 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.620112 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.620610 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.621576 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.621782 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.621840 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.622472 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:58.622473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:58.622473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:58.622478 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:58.622490 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:58.622494 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:58.622533 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:58.622583 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:12:58.622604 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:12:58.622846 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.618709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41857 (local address 127.0.98.1:58605)
0928 23:12:58.618882 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:58.618912 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:58.619357 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:58.619545 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:58.619548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:58.619555 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:58.620022 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.620026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.620680 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.620681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.621484 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.621487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.621851 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.621851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.621946 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.622363 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:58.622364 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:58.622389 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:58.622410 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:58.622532 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:58.622534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:58.622536 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:58.622578 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:12:58.622663 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260928 23:12:58.623273 9839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41857
I20260928 23:12:58.623354 9839 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:58.623489 9839 heartbeater.cc:511] Master 127.0.98.62:41857 requested a full tablet report, sending...
I20260928 23:12:58.624051 9651 ts_manager.cc:194] Registered new tserver with Master: 2938b7c26b784480a23c0ec9570241ad (127.0.98.1:40341)
I20260928 23:12:58.624601 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58605
I20260928 23:12:58.625787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9711
I20260928 23:12:58.625861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:58.626029 392 external_mini_cluster.cc:1428] Setting key 6aa1097c007abee510e9f70a9230a91e
I20260928 23:12:58.626770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--builtin_ntp_servers=127.0.98.20:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 23:12:58.706000 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:58.706202 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:58.706229 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:58.707357 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:58.707396 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:58.707427 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:58.708715 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:58.708943 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:58.709331 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:58.714076 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.714088 9842 server_base.cc:1034] running on GCE node
W20260928 23:12:58.714046 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:58.714054 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.714534 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:58.714758 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:58.715898 9842 hybrid_clock.cc:648] HybridClock initialized: now 1790637178715886 us; error 27 us; skew 500 ppm
I20260928 23:12:58.717105 9842 webserver.cc:460] Webserver started at http://127.0.98.2:37841/ using document root <none> and password file <none>
I20260928 23:12:58.717362 9842 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:58.717408 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:58.717536 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:58.718570 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2b853846caa40798ffb64771fd917bf"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "cf385d561403eaa84363f9eeeb4194fa"
server_key_iv: "9f6f0e10ee4c8d3d2bbe25e7ce9e92a6"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.718891 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2b853846caa40798ffb64771fd917bf"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "cf385d561403eaa84363f9eeeb4194fa"
server_key_iv: "9f6f0e10ee4c8d3d2bbe25e7ce9e92a6"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.720072 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:58.720667 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.720804 9842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:58.720883 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "d2b853846caa40798ffb64771fd917bf"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "cf385d561403eaa84363f9eeeb4194fa"
server_key_iv: "9f6f0e10ee4c8d3d2bbe25e7ce9e92a6"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.720935 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:58.740173 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:58.740445 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:58.740545 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:58.740728 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:58.741042 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:58.741073 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.741099 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:58.741117 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.746244 9842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39101
I20260928 23:12:58.746313 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39101 every 8 connection(s)
I20260928 23:12:58.746591 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:58.751060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9842
I20260928 23:12:58.751163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:58.751348 392 external_mini_cluster.cc:1428] Setting key e512777c3e29c0826949d3c4c16bbed0
I20260928 23:12:58.751677 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.747326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53535 (local address 127.0.98.62:41857)
0928 23:12:58.747389 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.747391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.747634 (+ 243us) server_negotiation.cc:407] Connection header received
0928 23:12:58.748110 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.748111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.748128 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.748181 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.748832 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.749362 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.750299 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.750482 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.750533 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.751393 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:58.751395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:58.751395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:58.751401 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:58.751418 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:58.751422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:58.751471 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:58.751568 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:12:58.751596 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:12:58.751888 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.747309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41857 (local address 127.0.98.2:53535)
0928 23:12:58.747503 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:58.747545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:58.748032 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:58.748247 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:58.748250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:58.748259 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:58.748771 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.748776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.749427 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.749428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.750223 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.750225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.750548 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.750548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.750668 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.751243 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:58.751243 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:58.751283 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:58.751310 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:58.751483 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:58.751489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:58.751492 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:58.751552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:12:58.751672 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 23:12:58.752254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--builtin_ntp_servers=127.0.98.20:40543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 23:12:58.752388 9970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41857
I20260928 23:12:58.752482 9970 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:58.752655 9970 heartbeater.cc:511] Master 127.0.98.62:41857 requested a full tablet report, sending...
I20260928 23:12:58.752997 9651 ts_manager.cc:194] Registered new tserver with Master: d2b853846caa40798ffb64771fd917bf (127.0.98.2:39101)
I20260928 23:12:58.753451 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53535
W20260928 23:12:58.833787 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:58.833979 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:58.833998 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:58.835057 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:58.835090 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:58.835120 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:58.836443 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:58.836658 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:58.836917 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:58.841616 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:58.841645 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:58.841693 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:58.841702 9973 server_base.cc:1034] running on GCE node
I20260928 23:12:58.842000 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:58.842226 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:58.843364 9973 hybrid_clock.cc:648] HybridClock initialized: now 1790637178843347 us; error 26 us; skew 500 ppm
I20260928 23:12:58.844487 9973 webserver.cc:460] Webserver started at http://127.0.98.3:41169/ using document root <none> and password file <none>
I20260928 23:12:58.844693 9973 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:58.844740 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:58.844863 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:58.845944 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "8026de48f1e64727a1f96c526e1c17df"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "f4c8c50032d7469b7a8b9e0fc6ad589c"
server_key_iv: "ba472d5f160146d2bdf99d7a8d596bd0"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.846278 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8026de48f1e64727a1f96c526e1c17df"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "f4c8c50032d7469b7a8b9e0fc6ad589c"
server_key_iv: "ba472d5f160146d2bdf99d7a8d596bd0"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.847419 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:12:58.848003 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.848145 9973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:58.848222 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "8026de48f1e64727a1f96c526e1c17df"
format_stamp: "Formatted at 2026-09-28 23:12:58 on dist-test-slave-6vgd"
server_key: "f4c8c50032d7469b7a8b9e0fc6ad589c"
server_key_iv: "ba472d5f160146d2bdf99d7a8d596bd0"
server_key_version: "encryptionkey@0"
I20260928 23:12:58.848285 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:58.862509 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:58.862766 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:58.862884 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:58.863073 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:58.863375 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:58.863408 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.863433 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:58.863451 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:58.868765 9973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43277
I20260928 23:12:58.868805 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43277 every 8 connection(s)
I20260928 23:12:58.869099 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:58.869701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 9973
I20260928 23:12:58.869792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:58.869987 392 external_mini_cluster.cc:1428] Setting key dee2ef2a18fd6cb150a1b425ec8772b6
I20260928 23:12:58.874140 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.869805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57835 (local address 127.0.98.62:41857)
0928 23:12:58.869858 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.869860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.870100 (+ 240us) server_negotiation.cc:407] Connection header received
0928 23:12:58.870719 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.870720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.870731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.870776 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.871432 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.871933 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.872862 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.873062 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.873115 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.873887 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:58.873888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:58.873889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:58.873893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:58.873907 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:58.873912 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:58.873956 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:58.874033 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:12:58.874062 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:12:58.874341 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:58.869761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41857 (local address 127.0.98.3:57835)
0928 23:12:58.869963 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:58.870006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:58.870634 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:58.870858 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:58.870862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:58.870872 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:58.871355 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.871359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.872015 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.872016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.872789 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:12:58.872790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:12:58.873160 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:12:58.873161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:12:58.873322 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:12:58.873771 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:58.873772 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:58.873799 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:58.873819 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:58.873962 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:58.873966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:58.873969 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:58.874035 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:12:58.874149 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 23:12:58.874727 10101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41857
I20260928 23:12:58.874796 10101 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:58.874953 10101 heartbeater.cc:511] Master 127.0.98.62:41857 requested a full tablet report, sending...
I20260928 23:12:58.875268 9651 ts_manager.cc:194] Registered new tserver with Master: 8026de48f1e64727a1f96c526e1c17df (127.0.98.3:43277)
I20260928 23:12:58.875622 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57835
I20260928 23:12:58.881546 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:12:58.883106 9691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:58.882721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.0.98.62:41857)
0928 23:12:58.882782 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:58.882784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:58.882844 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:12:58.882885 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:58.882885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:58.882895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:58.882938 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:58.883050 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35306: BlockingRecv error: recv got EOF from 127.0.0.1:35306 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:12:58.883176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9711
I20260928 23:12:58.887813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9842
I20260928 23:12:58.891479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9973
I20260928 23:12:58.895150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 9620
2026-09-28T23:12:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T23:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:12:58Z Disabled control of system clock
I20260928 23:12:58.919184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42363
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:12:58.996047 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:58.996268 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:58.996289 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:58.997478 10112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:12:58.997517 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:58.997529 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:58.997543 10112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:12:58.997566 10112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:12:58.998939 10112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:58.999186 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:58.999466 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:59.003886 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.003932 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.003886 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.004184 10112 server_base.cc:1034] running on GCE node
I20260928 23:12:59.004401 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:59.004654 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:59.005788 10112 hybrid_clock.cc:648] HybridClock initialized: now 1790637179005775 us; error 29 us; skew 500 ppm
I20260928 23:12:59.006979 10112 webserver.cc:460] Webserver started at http://127.0.98.62:46559/ using document root <none> and password file <none>
I20260928 23:12:59.007189 10112 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:59.007233 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:59.007351 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:59.008383 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "db799fc9b7ab41de801ab6a80865c96b"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "cec170865a4b3607394ec227d872d976"
server_key_iv: "80d45499ef8e9d080135da960f497977"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.008708 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "db799fc9b7ab41de801ab6a80865c96b"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "cec170865a4b3607394ec227d872d976"
server_key_iv: "80d45499ef8e9d080135da960f497977"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.009930 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:59.010536 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.010722 10112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:12:59.010810 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "db799fc9b7ab41de801ab6a80865c96b"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "cec170865a4b3607394ec227d872d976"
server_key_iv: "80d45499ef8e9d080135da960f497977"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.010869 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:59.014029 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:59.014262 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:59.014376 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:59.017830 10112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42363
I20260928 23:12:59.017860 10178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42363 every 8 connection(s)
I20260928 23:12:59.018141 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:12:59.018741 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:12:59.021308 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b: Bootstrap starting.
I20260928 23:12:59.021798 10179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b: Neither blocks nor log segments found. Creating new log.
I20260928 23:12:59.022017 10179 log.cc:824] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:12:59.022552 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b: No bootstrap required, opened a new log
I20260928 23:12:59.022691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10112
I20260928 23:12:59.022776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:12:59.022931 392 external_mini_cluster.cc:1428] Setting key e4eb5aac70611c2d1364e80df258f35c
I20260928 23:12:59.024469 10179 raft_consensus.cc:348] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260928 23:12:59.024591 10179 raft_consensus.cc:374] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:12:59.024611 10179 raft_consensus.cc:729] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db799fc9b7ab41de801ab6a80865c96b, State: Initialized, Role: FOLLOWER
I20260928 23:12:59.024775 10179 consensus_queue.cc:260] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260928 23:12:59.024873 10179 raft_consensus.cc:388] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:12:59.024904 10179 raft_consensus.cc:482] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:12:59.024948 10179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:12:59.025480 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.023398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.98.62:42363)
0928 23:12:59.023560 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:59.023566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:59.023589 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:12:59.024080 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:59.024086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:59.024121 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:59.024237 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:59.024872 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:59.024876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:59.024876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:59.024881 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:59.024925 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:59.024938 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:59.025002 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:59.025069 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:12:59.025122 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260928 23:12:59.025523 10179 raft_consensus.cc:504] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260928 23:12:59.025612 10179 leader_election.cc:302] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db799fc9b7ab41de801ab6a80865c96b; no voters:
I20260928 23:12:59.025768 10179 leader_election.cc:288] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:12:59.025804 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:12:59.025943 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [term 1 LEADER]: Becoming Leader. State: Replica: db799fc9b7ab41de801ab6a80865c96b, State: Running, Role: LEADER
I20260928 23:12:59.025993 10179 sys_catalog.cc:564] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:12:59.026067 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } }
I20260928 23:12:59.026417 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader db799fc9b7ab41de801ab6a80865c96b. Latest consensus state: current_term: 1 leader_uuid: "db799fc9b7ab41de801ab6a80865c96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } } }
I20260928 23:12:59.026439 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db799fc9b7ab41de801ab6a80865c96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db799fc9b7ab41de801ab6a80865c96b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42363 } } }
I20260928 23:12:59.026520 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:59.026580 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b [sys.catalog]: This master's current role is: LEADER
I20260928 23:12:59.026763 10193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:12:59.027233 10193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:12:59.028648 10193 catalog_manager.cc:1347] Generated new cluster ID: 7570c78d60314abcb777ab95a9deb7a5
I20260928 23:12:59.028698 10193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:12:59.037384 10193 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:12:59.037523 10193 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:12:59.041524 10193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db799fc9b7ab41de801ab6a80865c96b: Generated new TSK 0
I20260928 23:12:59.041688 10193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:12:59.043344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 23:12:59.120085 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:59.120276 10203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:59.120296 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:59.121531 10203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:59.121608 10203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:59.121668 10203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:12:59.123073 10203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:59.123281 10203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:59.123550 10203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:59.128280 10209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.128314 10211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.128280 10208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.128543 10203 server_base.cc:1034] running on GCE node
I20260928 23:12:59.128731 10203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:59.128947 10203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:59.130084 10203 hybrid_clock.cc:648] HybridClock initialized: now 1790637179130073 us; error 27 us; skew 500 ppm
I20260928 23:12:59.131273 10203 webserver.cc:460] Webserver started at http://127.0.98.1:41087/ using document root <none> and password file <none>
I20260928 23:12:59.131480 10203 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:59.131522 10203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:59.131640 10203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:59.132669 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "635b5270560b42ebb38b8ed388fd8aed"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "e89126843dd23f9bf510b4811f67afc4"
server_key_iv: "eaedab23d56bcd8119dc4ee98dce16ac"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.132995 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "635b5270560b42ebb38b8ed388fd8aed"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "e89126843dd23f9bf510b4811f67afc4"
server_key_iv: "eaedab23d56bcd8119dc4ee98dce16ac"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.134272 10203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:59.134959 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.135102 10203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:59.135176 10203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "635b5270560b42ebb38b8ed388fd8aed"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "e89126843dd23f9bf510b4811f67afc4"
server_key_iv: "eaedab23d56bcd8119dc4ee98dce16ac"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.135247 10203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:59.138536 10203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:59.138793 10203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:59.138909 10203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:59.139099 10203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:59.139417 10203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:59.139454 10203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.139480 10203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:59.139503 10203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.145849 10203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37029
I20260928 23:12:59.145922 10330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37029 every 8 connection(s)
I20260928 23:12:59.146214 10203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:12:59.147790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10203
I20260928 23:12:59.147869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:12:59.148062 392 external_mini_cluster.cc:1428] Setting key c2bb0cae17f815b1df3a9eab354d85ee
I20260928 23:12:59.149044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:12:59.149240 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.146864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35265 (local address 127.0.98.62:42363)
0928 23:12:59.146906 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:59.146907 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:59.147141 (+ 234us) server_negotiation.cc:407] Connection header received
0928 23:12:59.147623 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:59.147624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:59.147633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:59.147673 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:59.148977 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:59.148978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:59.148979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:59.148983 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:59.148999 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:59.149004 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:59.149069 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:59.149129 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 23:12:59.149157 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:12:59.149421 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.146849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.1:35265)
0928 23:12:59.147016 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:59.147049 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:59.147509 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:59.147752 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:59.147757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:59.147766 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:59.148827 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:59.148827 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:59.148864 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:59.148892 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:59.149068 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:59.149071 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:59.149073 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:59.149128 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 23:12:59.149195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260928 23:12:59.149855 10331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260928 23:12:59.149928 10331 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:59.150063 10331 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260928 23:12:59.150552 10143 ts_manager.cc:194] Registered new tserver with Master: 635b5270560b42ebb38b8ed388fd8aed (127.0.98.1:37029)
W20260928 23:12:59.227490 10334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:59.227708 10334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:59.227741 10334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:59.228883 10334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:59.228920 10334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:59.228953 10334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:12:59.230278 10334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:59.230495 10334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:59.230756 10334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:59.235445 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.235471 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.235505 10334 server_base.cc:1034] running on GCE node
W20260928 23:12:59.235605 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.235880 10334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:59.236100 10334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:59.237252 10334 hybrid_clock.cc:648] HybridClock initialized: now 1790637179237234 us; error 36 us; skew 500 ppm
I20260928 23:12:59.238514 10334 webserver.cc:460] Webserver started at http://127.0.98.2:40253/ using document root <none> and password file <none>
I20260928 23:12:59.238763 10334 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:59.238812 10334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:59.238934 10334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:59.239976 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "f99a5ee3ec53463d9ccae70e74e8eec6"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "dfe981cbf3f25a89eb628077f95d23fd"
server_key_iv: "0275b98c85da8ca5d2aad6a4de26af8d"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.240341 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f99a5ee3ec53463d9ccae70e74e8eec6"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "dfe981cbf3f25a89eb628077f95d23fd"
server_key_iv: "0275b98c85da8ca5d2aad6a4de26af8d"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.241585 10334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:12:59.242236 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.242365 10334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:12:59.242444 10334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "f99a5ee3ec53463d9ccae70e74e8eec6"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "dfe981cbf3f25a89eb628077f95d23fd"
server_key_iv: "0275b98c85da8ca5d2aad6a4de26af8d"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.242504 10334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:59.245788 10334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:59.246021 10334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:59.246140 10334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:59.246336 10334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:59.246604 10334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:59.246635 10334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.246660 10334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:59.246677 10334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.251879 10334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35815
I20260928 23:12:59.251917 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35815 every 8 connection(s)
I20260928 23:12:59.252229 10334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:12:59.253383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10334
I20260928 23:12:59.253482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:12:59.253646 392 external_mini_cluster.cc:1428] Setting key f5c3abe1d9d870a3c148aa5dd37709d7
I20260928 23:12:59.254482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--builtin_ntp_servers=127.0.98.20:46465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 23:12:59.255275 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.252957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37957 (local address 127.0.98.62:42363)
0928 23:12:59.252996 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:59.252998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:59.253254 (+ 256us) server_negotiation.cc:407] Connection header received
0928 23:12:59.253778 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:59.253780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:59.253787 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:59.253843 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:59.255039 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:59.255041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:59.255041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:59.255046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:59.255063 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:59.255068 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:59.255129 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:59.255195 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 23:12:59.255209 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:12:59.255488 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.252939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.2:37957)
0928 23:12:59.253106 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:59.253138 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:59.253674 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:59.253925 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:59.253930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:59.253940 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:59.254897 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:59.254897 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:59.254926 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:59.254953 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:59.255142 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:59.255145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:59.255147 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:12:59.255201 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 23:12:59.255248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 23:12:59.255995 10462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260928 23:12:59.256078 10462 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:59.256194 10462 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260928 23:12:59.256508 10143 ts_manager.cc:194] Registered new tserver with Master: f99a5ee3ec53463d9ccae70e74e8eec6 (127.0.98.2:35815)
W20260928 23:12:59.334090 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:12:59.334322 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:12:59.334345 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:12:59.335501 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:12:59.335538 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:12:59.335564 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:12:59.336894 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:12:59.337110 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:12:59.337432 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:12:59.342492 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.342605 10465 server_base.cc:1034] running on GCE node
W20260928 23:12:59.342432 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:12:59.342435 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:12:59.342926 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:12:59.343166 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:12:59.344321 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790637179344310 us; error 30 us; skew 500 ppm
I20260928 23:12:59.345595 10465 webserver.cc:460] Webserver started at http://127.0.98.3:35663/ using document root <none> and password file <none>
I20260928 23:12:59.345830 10465 fs_manager.cc:359] Metadata directory not provided
I20260928 23:12:59.345881 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:12:59.346014 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:12:59.347074 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "f21f466e3a1b49cfb38491de03848377"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "a67f85c80d8033ae88b7c7926c16592f"
server_key_iv: "1c98dbd9b78827b353044a293d081fb8"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.347404 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f21f466e3a1b49cfb38491de03848377"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "a67f85c80d8033ae88b7c7926c16592f"
server_key_iv: "1c98dbd9b78827b353044a293d081fb8"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.348611 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:12:59.349319 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.349466 10465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.349545 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "f21f466e3a1b49cfb38491de03848377"
format_stamp: "Formatted at 2026-09-28 23:12:59 on dist-test-slave-6vgd"
server_key: "a67f85c80d8033ae88b7c7926c16592f"
server_key_iv: "1c98dbd9b78827b353044a293d081fb8"
server_key_version: "encryptionkey@0"
I20260928 23:12:59.349609 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:12:59.352816 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:12:59.353077 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:12:59.353199 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:12:59.353430 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:12:59.353719 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:12:59.353753 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.353780 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:12:59.353803 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:12:59.359124 10465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37307
I20260928 23:12:59.359171 10592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37307 every 8 connection(s)
I20260928 23:12:59.359483 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:12:59.362645 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.360152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44345 (local address 127.0.98.62:42363)
0928 23:12:59.360199 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:59.360201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:59.360449 (+ 248us) server_negotiation.cc:407] Connection header received
0928 23:12:59.360946 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:59.360947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:59.360954 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:59.361006 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:59.362394 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:12:59.362396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:12:59.362396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:12:59.362401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:12:59.362417 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:12:59.362421 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:12:59.362471 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:12:59.362561 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 23:12:59.362575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:12:59.362845 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:12:59.360139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42363 (local address 127.0.98.3:44345)
0928 23:12:59.360332 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 23:12:59.360367 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:12:59.360846 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:12:59.361080 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:12:59.361084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:12:59.361094 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:12:59.362202 (+ 1108us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:12:59.362203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:12:59.362242 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:12:59.362281 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:12:59.362486 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:12:59.362491 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:12:59.362494 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:12:59.362572 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 23:12:59.362635 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260928 23:12:59.363411 10593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42363
I20260928 23:12:59.363524 10593 heartbeater.cc:464] Registering TS with master...
I20260928 23:12:59.363682 10593 heartbeater.cc:511] Master 127.0.98.62:42363 requested a full tablet report, sending...
I20260928 23:12:59.364037 10143 ts_manager.cc:194] Registered new tserver with Master: f21f466e3a1b49cfb38491de03848377 (127.0.98.3:37307)
I20260928 23:12:59.369419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10465
I20260928 23:12:59.369500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:12:59.369647 392 external_mini_cluster.cc:1428] Setting key 8c55afe227aa1984a29dedb8463c7305
I20260928 23:12:59.370464 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 23:12:59.372107 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:12:59.371704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.0.98.62:42363)
0928 23:12:59.371752 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:12:59.371754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:12:59.371853 (+ 99us) server_negotiation.cc:407] Connection header received
0928 23:12:59.371899 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:12:59.371899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:12:59.371904 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:12:59.371955 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:12:59.372050 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51928: BlockingRecv error: recv got EOF from 127.0.0.1:51928 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:12:59.372191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10203
I20260928 23:12:59.376724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10334
I20260928 23:12:59.380499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10465
I20260928 23:12:59.384408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10112
2026-09-28T23:12:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (489 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:12:59 dist-test-slave-6vgd krb5kdc[10603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:12:59 dist-test-slave-6vgd krb5kdc[10603](info): set up 2 sockets
Sep 28 23:12:59 dist-test-slave-6vgd krb5kdc[10603](info): commencing operation
krb5kdc: starting...
W20260928 23:13:00.321285 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:13:00.430294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:00.508586 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:00.508796 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:00.508816 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:00.509904 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:13:00.509938 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:00.509951 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:00.509974 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:13:00.509984 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:13:00.511340 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:00.511520 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:00.511689 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:00.516083 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.516183 10616 server_base.cc:1034] running on GCE node
W20260928 23:13:00.516165 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:00.516129 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.516539 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:00.516799 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:00.517935 10616 hybrid_clock.cc:648] HybridClock initialized: now 1790637180517918 us; error 34 us; skew 500 ppm
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:00.520124 10616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:13:00.520532 10616 webserver.cc:460] Webserver started at http://127.0.98.62:34033/ using document root <none> and password file <none>
I20260928 23:13:00.520717 10616 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:00.520761 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:00.520881 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:13:00.521903 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "6236a410145447a7830ea5b867021c87"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "480787dd845b62343659a4e1c6ec1b18"
server_key_iv: "134390026a5a05328edbd720c002474c"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.522219 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "6236a410145447a7830ea5b867021c87"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "480787dd845b62343659a4e1c6ec1b18"
server_key_iv: "134390026a5a05328edbd720c002474c"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.523378 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:13:00.523993 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.524112 10616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:13:00.524189 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "6236a410145447a7830ea5b867021c87"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "480787dd845b62343659a4e1c6ec1b18"
server_key_iv: "134390026a5a05328edbd720c002474c"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.524247 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:00.535763 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:00.537343 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:00.537468 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:00.540923 10616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38099
I20260928 23:13:00.540966 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38099 every 8 connection(s)
I20260928 23:13:00.541301 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:13:00.542233 10684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:00.544214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10616
I20260928 23:13:00.544296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:13:00.544407 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: Bootstrap starting.
I20260928 23:13:00.544462 392 external_mini_cluster.cc:1428] Setting key 622dadf7ae71481e1c738ecbecc63132
I20260928 23:13:00.545043 10684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:00.545351 10684 log.cc:824] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:00.546015 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: No bootstrap required, opened a new log
I20260928 23:13:00.548036 10684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260928 23:13:00.548202 10684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:00.548233 10684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6236a410145447a7830ea5b867021c87, State: Initialized, Role: FOLLOWER
I20260928 23:13:00.548388 10684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260928 23:13:00.548473 10684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:13:00.548502 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:13:00.548550 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:00.549160 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260928 23:13:00.549278 10684 leader_election.cc:302] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6236a410145447a7830ea5b867021c87; no voters:
I20260928 23:13:00.549428 10684 leader_election.cc:288] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:13:00.549460 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:13:00.549633 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [term 1 LEADER]: Becoming Leader. State: Replica: 6236a410145447a7830ea5b867021c87, State: Running, Role: LEADER
I20260928 23:13:00.549708 10684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:13:00.549722 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } }
I20260928 23:13:00.550071 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6236a410145447a7830ea5b867021c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260928 23:13:00.550205 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [sys.catalog]: This master's current role is: LEADER
I20260928 23:13:00.550374 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6236a410145447a7830ea5b867021c87. Latest consensus state: current_term: 1 leader_uuid: "6236a410145447a7830ea5b867021c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6236a410145447a7830ea5b867021c87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38099 } } }
I20260928 23:13:00.550477 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87 [sys.catalog]: This master's current role is: LEADER
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260928 23:13:00.551414 10705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:13:00.551453 10705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:13:00.551553 10697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:13:00.551774 10697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:13:00.554116 10697 catalog_manager.cc:1347] Generated new cluster ID: e07b40a738824545940d1347f7771923
I20260928 23:13:00.554214 10697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:13:00.555131 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:00.544930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.0.98.62:38099)
0928 23:13:00.545089 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:00.545097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:00.545126 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:13:00.545618 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:00.545625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:00.545674 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:00.545784 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:00.546470 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.547311 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.548354 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.548796 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.548875 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:00.551626 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:00.551630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:00.551630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:00.551642 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:00.553525 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.553903 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.553906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.553906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.554013 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.554207 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.554209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.554209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.554548 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:00.554719 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:00.554791 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:13:00.554926 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260928 23:13:00.563774 10697 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:13:00.564303 10697 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:13:00.573123 10697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: Generated new TSK 0
I20260928 23:13:00.573352 10697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:13:00.597018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:00.675261 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:00.675473 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:00.675495 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:00.676643 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:00.676690 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:00.676715 10710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:13:00.678139 10710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:00.678344 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:00.678532 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:00.683293 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:00.683301 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.683470 10710 server_base.cc:1034] running on GCE node
W20260928 23:13:00.683320 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.683703 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:00.683921 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:00.685053 10710 hybrid_clock.cc:648] HybridClock initialized: now 1790637180685044 us; error 30 us; skew 500 ppm
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:00.687371 10710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:13:00.687749 10710 webserver.cc:460] Webserver started at http://127.0.98.1:39655/ using document root <none> and password file <none>
I20260928 23:13:00.687943 10710 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:00.687983 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:00.688071 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:13:00.689044 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "b39bcb1f94e54d42b502442629cfad16"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "669551238a490bc16ab371105fa70587"
server_key_iv: "c1f0a48ac439468fdae27e83f49eaef5"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.689378 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b39bcb1f94e54d42b502442629cfad16"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "669551238a490bc16ab371105fa70587"
server_key_iv: "c1f0a48ac439468fdae27e83f49eaef5"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.690523 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:13:00.691179 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.691323 10710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:13:00.691391 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "b39bcb1f94e54d42b502442629cfad16"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "669551238a490bc16ab371105fa70587"
server_key_iv: "c1f0a48ac439468fdae27e83f49eaef5"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.691457 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:00.708920 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:00.711143 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:00.711277 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:00.711503 10710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:00.711800 10710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:00.711843 10710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.711865 10710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:00.711876 10710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.717286 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35021
I20260928 23:13:00.717341 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35021 every 8 connection(s)
I20260928 23:13:00.717633 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:13:00.721431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10710
I20260928 23:13:00.721534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:13:00.721722 392 external_mini_cluster.cc:1428] Setting key 4cbf7b09a06321eb40995b3a758d2fad
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:00.726984 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:00.718295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52257 (local address 127.0.98.62:38099)
0928 23:13:00.718381 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:00.718383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:00.718548 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:13:00.719027 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:00.719029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:00.719038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:00.719098 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:00.720011 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.720537 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.721634 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.721866 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.721934 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:00.724269 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:00.724272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:00.724272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:00.724280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:00.725697 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.726055 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.726056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.726056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.726115 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.726326 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.726327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.726327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.726453 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:00.726508 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:00.726801 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 23:13:00.726894 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 23:13:00.727149 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:00.718266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.1:52257)
0928 23:13:00.718427 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:00.718470 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:00.718944 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:00.719166 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:00.719170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:00.719717 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:00.719925 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:00.719929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.720605 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.720606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:00.721559 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:00.721562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.721962 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.721963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:00.722108 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:00.722618 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:00.722628 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:00.724178 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:00.725768 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:00.725774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:00.725777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:00.725991 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:00.726168 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:00.726168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:00.726169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:00.726273 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:00.726507 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:00.726514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:00.726671 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 23:13:00.726778 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 23:13:00.726937 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260928 23:13:00.727596 10839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260928 23:13:00.727680 10839 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:00.727881 10839 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260928 23:13:00.728385 10648 ts_manager.cc:194] Registered new tserver with Master: b39bcb1f94e54d42b502442629cfad16 (127.0.98.1:35021)
I20260928 23:13:00.728998 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52257
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:13:00.744374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:00.825165 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:00.825415 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:00.825448 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:00.826653 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:00.826700 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:00.826736 10844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:13:00.828261 10844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:00.828517 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:00.828725 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:00.833448 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:00.833491 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.833563 10844 server_base.cc:1034] running on GCE node
W20260928 23:13:00.833447 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.833889 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:00.834124 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:00.835271 10844 hybrid_clock.cc:648] HybridClock initialized: now 1790637180835260 us; error 26 us; skew 500 ppm
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:00.837646 10844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:13:00.838078 10844 webserver.cc:460] Webserver started at http://127.0.98.2:32875/ using document root <none> and password file <none>
I20260928 23:13:00.838289 10844 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:00.838335 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:00.838464 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:13:00.839480 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "4aa3754841ae4e0e8ef163c9d6021cca"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "543f58fcca50cd4570b906b5c59de023"
server_key_iv: "cb18ad5ea3295beb548a46da683b6622"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.839812 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4aa3754841ae4e0e8ef163c9d6021cca"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "543f58fcca50cd4570b906b5c59de023"
server_key_iv: "cb18ad5ea3295beb548a46da683b6622"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.841068 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:13:00.841811 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.841930 10844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.842011 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "4aa3754841ae4e0e8ef163c9d6021cca"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "543f58fcca50cd4570b906b5c59de023"
server_key_iv: "cb18ad5ea3295beb548a46da683b6622"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.842078 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:00.856837 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:00.858628 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:00.858752 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:00.858968 10844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:00.859246 10844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:00.859277 10844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.859298 10844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:00.859321 10844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.864640 10844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46221
I20260928 23:13:00.864696 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46221 every 8 connection(s)
I20260928 23:13:00.864980 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:13:00.868647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10844
I20260928 23:13:00.868757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:13:00.868952 392 external_mini_cluster.cc:1428] Setting key 7e1572d6e07ae76f5a932c9fefb7ca09
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:00.874609 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:00.865679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50737 (local address 127.0.98.62:38099)
0928 23:13:00.865722 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:00.865723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:00.865921 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:13:00.866438 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:00.866439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:00.866450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:00.866494 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:00.867413 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.867902 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.869095 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.869371 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.869423 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:00.871792 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:00.871793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:00.871793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:00.871800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:00.873113 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.873502 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.873504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.873505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.873564 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:00.873926 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:00.873927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:00.873927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:00.874067 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:00.874139 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:00.874435 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 23:13:00.874515 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:13:00.874717 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:00.865644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.2:50737)
0928 23:13:00.865804 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:00.865843 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:00.866343 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:00.866574 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:00.866577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:00.867124 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:00.867324 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:00.867329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.867981 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.867982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:00.869008 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:00.869011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:00.869443 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:00.869444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:00.869578 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:00.870036 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:00.870046 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:00.871705 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:00.873195 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:00.873201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:00.873203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:00.873434 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:00.873736 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:00.873737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:00.873737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:00.873854 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:00.874153 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:00.874160 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:00.874284 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:13:00.874400 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 23:13:00.874532 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260928 23:13:00.875242 10973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260928 23:13:00.875347 10973 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:00.875524 10973 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260928 23:13:00.875864 10648 ts_manager.cc:194] Registered new tserver with Master: 4aa3754841ae4e0e8ef163c9d6021cca (127.0.98.2:46221)
I20260928 23:13:00.876209 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50737
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:13:00.892411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:00.971205 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:00.971441 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:00.971475 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:00.972640 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:00.972687 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:00.972721 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:13:00.974234 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:00.974486 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:00.974705 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:00.979450 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:00.979450 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:00.979429 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:00.979704 10978 server_base.cc:1034] running on GCE node
I20260928 23:13:00.979892 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:00.980114 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:00.981264 10978 hybrid_clock.cc:648] HybridClock initialized: now 1790637180981242 us; error 35 us; skew 500 ppm
Sep 28 23:13:00 dist-test-slave-6vgd krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:00.983695 10978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:13:00.984123 10978 webserver.cc:460] Webserver started at http://127.0.98.3:39247/ using document root <none> and password file <none>
I20260928 23:13:00.984313 10978 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:00.984364 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:00.984486 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:13:00.985575 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "d024bb9b167fda297f9275089df58b90"
server_key_iv: "39ce15f17a36dca9cc58dfd43f324303"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.985893 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "d024bb9b167fda297f9275089df58b90"
server_key_iv: "39ce15f17a36dca9cc58dfd43f324303"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.987093 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:13:00.987807 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:00.987941 10978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:13:00.988013 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d"
format_stamp: "Formatted at 2026-09-28 23:13:00 on dist-test-slave-6vgd"
server_key: "d024bb9b167fda297f9275089df58b90"
server_key_iv: "39ce15f17a36dca9cc58dfd43f324303"
server_key_version: "encryptionkey@0"
I20260928 23:13:00.988077 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:01.006764 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:01.009110 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:01.009271 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:01.009478 10978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:01.009742 10978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:01.009783 10978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:01.009804 10978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:01.009820 10978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:01.015128 10978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41955
I20260928 23:13:01.015168 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41955 every 8 connection(s)
I20260928 23:13:01.015466 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:13:01.017098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 10978
I20260928 23:13:01.017180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:13:01.017369 392 external_mini_cluster.cc:1428] Setting key fa0e91b13c55f00355b85f22b7dfa1ba
Sep 28 23:13:01 dist-test-slave-6vgd krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:01.024156 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.016207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47097 (local address 127.0.98.62:38099)
0928 23:13:01.016286 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.016289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.016411 (+ 122us) server_negotiation.cc:407] Connection header received
0928 23:13:01.016897 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.016898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.016909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.016985 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:01.018083 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.018613 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.019593 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.019790 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.019842 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.021854 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:01.021855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:01.021855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:01.021863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:01.022954 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:01.023301 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:01.023302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:01.023302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:01.023355 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:01.023553 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:01.023554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:01.023554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:01.023661 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:01.023715 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.023982 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.024068 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 23:13:01.024336 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.016089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38099 (local address 127.0.98.3:47097)
0928 23:13:01.016242 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.016326 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.016808 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.017055 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.017058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.017727 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:01.018001 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.018007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.018693 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.018694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.019509 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.019511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.019853 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.019854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.019979 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.020462 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:01.020470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:01.021769 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:01.023021 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:01.023027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:01.023029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:01.023238 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:01.023406 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:01.023406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:01.023407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:01.023500 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:01.023712 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:01.023717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:01.023848 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.023961 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.024111 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 23:13:01.024701 11107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38099
I20260928 23:13:01.024791 11107 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:01.024957 11107 heartbeater.cc:511] Master 127.0.98.62:38099 requested a full tablet report, sending...
I20260928 23:13:01.025295 10648 ts_manager.cc:194] Registered new tserver with Master: 3db16c7c878e4ea3a7b001c6dcec5c4d (127.0.98.3:41955)
I20260928 23:13:01.025650 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47097
I20260928 23:13:01.028903 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:13:01.035585 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.030156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.0.98.62:38099)
0928 23:13:01.030208 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.030210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.030326 (+ 116us) server_negotiation.cc:407] Connection header received
0928 23:13:01.030360 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.030360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.030368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.030410 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:01.030928 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.031417 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.032127 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.032335 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.032388 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.033536 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:01.033537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:01.033537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:01.033543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:01.034664 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:01.035010 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:01.035011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:01.035011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:01.035063 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:01.035198 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:01.035198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:01.035198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:01.035321 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:01.035404 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.035466 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.035520 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:13:01.037039 10648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:13:01.038578 10648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:13:01.044313 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.0.1:45512)
0928 23:13:01.040775 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.040784 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.040833 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.041101 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.041102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.041104 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:01.041163 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.041164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042123 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.042124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.043276 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.043278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.044034 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.044110 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044123 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.044186 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.044237 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":63,"threads_started":1}
I20260928 23:13:01.044323 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.2:46221)
0928 23:13:01.040793 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.040797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.040892 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:13:01.040937 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.040940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.040954 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.041027 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:01.041256 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042038 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.043368 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.043965 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044025 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044084 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.044205 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.044241 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":76,"threads_started":1}
I20260928 23:13:01.044613 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.0.98.1:35021)
0928 23:13:01.041010 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.041014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.041057 (+ 43us) server_negotiation.cc:407] Connection header received
0928 23:13:01.041122 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.041124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.041138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.041231 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:01.041472 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042506 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.043444 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.044043 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044107 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044159 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.044494 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.044534 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 23:13:01.044427 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.0.1:41006)
0928 23:13:01.041005 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.041014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.041064 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.041273 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.041274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.041275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:01.041311 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.041312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042575 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.042576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.043353 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.043355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.044192 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.044253 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044258 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.044303 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.044343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":51,"threads_started":1}
I20260928 23:13:01.044782 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41955 (local address 127.0.0.1:44872)
0928 23:13:01.040648 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.040691 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.040751 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.041085 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.041088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.041090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:01.041156 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.041158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042532 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.042533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.043598 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.043599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.044472 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.044530 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044533 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.044644 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.044683 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":9,"thread_start_us":59,"threads_started":1}
I20260928 23:13:01.044922 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.040573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.0.98.3:41955)
0928 23:13:01.040872 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.040875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.040884 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:13:01.040913 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.040914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.040928 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.040999 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:01.041322 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.042431 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.043675 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.044339 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.044392 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.044441 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.044790 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.044836 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":89,"threads_started":1}
I20260928 23:13:01.045367 10907 tablet_service.cc:1467] Processing CreateTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 (DEFAULT_TABLE table=test-table [id=af8909f8a1024db0a6df18a90f294fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:01.045496 11041 tablet_service.cc:1467] Processing CreateTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 (DEFAULT_TABLE table=test-table [id=af8909f8a1024db0a6df18a90f294fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:01.045715 10907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5923b6e34b15420888b60fbd3342d6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:01.045797 11041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5923b6e34b15420888b60fbd3342d6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:01.048106 11121 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Bootstrap starting.
I20260928 23:13:01.048106 11122 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: Bootstrap starting.
I20260928 23:13:01.048755 11121 tablet_bootstrap.cc:654] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:01.048873 11122 tablet_bootstrap.cc:654] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:01.048928 10773 tablet_service.cc:1467] Processing CreateTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 (DEFAULT_TABLE table=test-table [id=af8909f8a1024db0a6df18a90f294fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:01.049279 11121 log.cc:824] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:01.049355 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5923b6e34b15420888b60fbd3342d6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:01.049539 11122 log.cc:824] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:01.050035 11121 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: No bootstrap required, opened a new log
I20260928 23:13:01.050101 11121 ts_tablet_manager.cc:1397] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:13:01.050313 11122 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: No bootstrap required, opened a new log
I20260928 23:13:01.050380 11122 ts_tablet_manager.cc:1397] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:13:01.051581 11121 raft_consensus.cc:348] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.051802 11121 raft_consensus.cc:374] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:01.051889 11121 raft_consensus.cc:729] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db16c7c878e4ea3a7b001c6dcec5c4d, State: Initialized, Role: FOLLOWER
I20260928 23:13:01.052080 11121 consensus_queue.cc:260] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.052342 11121 ts_tablet_manager.cc:1428] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:13:01.052464 11107 heartbeater.cc:503] Master 127.0.98.62:38099 was elected leader, sending a full tablet report...
I20260928 23:13:01.052383 11122 raft_consensus.cc:348] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.052542 11122 raft_consensus.cc:374] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:01.052577 11122 raft_consensus.cc:729] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa3754841ae4e0e8ef163c9d6021cca, State: Initialized, Role: FOLLOWER
I20260928 23:13:01.052770 11122 consensus_queue.cc:260] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.052856 11126 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: Bootstrap starting.
I20260928 23:13:01.053061 10973 heartbeater.cc:503] Master 127.0.98.62:38099 was elected leader, sending a full tablet report...
I20260928 23:13:01.053048 11122 ts_tablet_manager.cc:1428] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 23:13:01.053547 11126 tablet_bootstrap.cc:654] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:01.054041 11126 log.cc:824] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:01.054669 11126 tablet_bootstrap.cc:492] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: No bootstrap required, opened a new log
I20260928 23:13:01.054719 11126 ts_tablet_manager.cc:1397] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:13:01.056020 11126 raft_consensus.cc:348] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.056164 11126 raft_consensus.cc:374] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:01.056233 11126 raft_consensus.cc:729] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39bcb1f94e54d42b502442629cfad16, State: Initialized, Role: FOLLOWER
I20260928 23:13:01.056354 11126 consensus_queue.cc:260] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.056571 11126 ts_tablet_manager.cc:1428] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:13:01.056617 10839 heartbeater.cc:503] Master 127.0.98.62:38099 was elected leader, sending a full tablet report...
I20260928 23:13:01.074378 11125 raft_consensus.cc:482] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:13:01.074496 11125 raft_consensus.cc:504] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.074795 11125 leader_election.cc:288] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b39bcb1f94e54d42b502442629cfad16 (127.0.98.1:35021), 4aa3754841ae4e0e8ef163c9d6021cca (127.0.98.2:46221)
I20260928 23:13:01.077544 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.075020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56127 (local address 127.0.98.1:35021)
0928 23:13:01.075057 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.075059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.075068 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:13:01.075150 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.075151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.075162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.075217 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:01.075372 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.075962 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.076844 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.077205 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.077300 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.077311 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.077444 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.077480 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:13:01.077571 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.074967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.98.3:56127)
0928 23:13:01.075024 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.075035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.075077 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.075230 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.075231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.075233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:01.075297 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.075299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.076028 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.076029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.076765 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.076766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.077299 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.077300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.077384 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.077387 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.077430 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.077472 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 23:13:01.077586 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.075045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40457 (local address 127.0.98.2:46221)
0928 23:13:01.075090 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.075092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.075209 (+ 117us) server_negotiation.cc:407] Connection header received
0928 23:13:01.075239 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.075241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.075251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.075298 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:01.075443 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.076004 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.076879 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.077288 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.077346 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.077355 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.077486 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.077520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:13:01.077801 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.074990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.98.3:40457)
0928 23:13:01.075139 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.075148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:01.075197 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:01.075306 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:01.075307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:01.075308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:01.075373 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.075375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.076072 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.076073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.076800 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:01.076802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.077352 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.077353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:01.077429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.077432 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:13:01.077476 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:13:01.077515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260928 23:13:01.077966 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5923b6e34b15420888b60fbd3342d6f9" candidate_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39bcb1f94e54d42b502442629cfad16" is_pre_election: true
I20260928 23:13:01.078123 10793 raft_consensus.cc:2454] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db16c7c878e4ea3a7b001c6dcec5c4d in term 0.
I20260928 23:13:01.078104 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5923b6e34b15420888b60fbd3342d6f9" candidate_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" is_pre_election: true
I20260928 23:13:01.078251 10927 raft_consensus.cc:2454] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db16c7c878e4ea3a7b001c6dcec5c4d in term 0.
I20260928 23:13:01.078336 10994 leader_election.cc:302] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db16c7c878e4ea3a7b001c6dcec5c4d, b39bcb1f94e54d42b502442629cfad16; no voters:
I20260928 23:13:01.078475 11125 raft_consensus.cc:2781] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:13:01.078541 11125 raft_consensus.cc:482] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:13:01.078557 11125 raft_consensus.cc:3037] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:01.079210 11125 raft_consensus.cc:504] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.079326 11125 leader_election.cc:288] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [CANDIDATE]: Term 1 election: Requested vote from peers b39bcb1f94e54d42b502442629cfad16 (127.0.98.1:35021), 4aa3754841ae4e0e8ef163c9d6021cca (127.0.98.2:46221)
I20260928 23:13:01.079449 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5923b6e34b15420888b60fbd3342d6f9" candidate_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aa3754841ae4e0e8ef163c9d6021cca"
I20260928 23:13:01.079493 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5923b6e34b15420888b60fbd3342d6f9" candidate_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b39bcb1f94e54d42b502442629cfad16"
I20260928 23:13:01.079523 10927 raft_consensus.cc:3037] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:01.079564 10793 raft_consensus.cc:3037] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:01.080363 10927 raft_consensus.cc:2454] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db16c7c878e4ea3a7b001c6dcec5c4d in term 1.
I20260928 23:13:01.080367 10793 raft_consensus.cc:2454] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db16c7c878e4ea3a7b001c6dcec5c4d in term 1.
I20260928 23:13:01.080519 10994 leader_election.cc:302] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3db16c7c878e4ea3a7b001c6dcec5c4d, b39bcb1f94e54d42b502442629cfad16; no voters:
I20260928 23:13:01.080590 11125 raft_consensus.cc:2781] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:13:01.080729 11125 raft_consensus.cc:686] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 1 LEADER]: Becoming Leader. State: Replica: 3db16c7c878e4ea3a7b001c6dcec5c4d, State: Running, Role: LEADER
I20260928 23:13:01.080806 11125 consensus_queue.cc:237] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } }
I20260928 23:13:01.081540 10647 catalog_manager.cc:5495] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3db16c7c878e4ea3a7b001c6dcec5c4d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3db16c7c878e4ea3a7b001c6dcec5c4d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 } health_report { overall_health: UNKNOWN } } }
I20260928 23:13:01.088964 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.086727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.98.3:41955)
0928 23:13:01.086762 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:01.086764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:01.086771 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:13:01.086838 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:01.086839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:01.086857 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:01.086927 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:13:01.087038 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.087494 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.088277 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:01.088476 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:01.088528 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:01.088665 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:13:01.088760 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:13:01.088825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:01.088860 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 23:13:01.088879 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:13:01.090562 10793 raft_consensus.cc:1261] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16 [term 1 FOLLOWER]: Refusing update from remote peer 3db16c7c878e4ea3a7b001c6dcec5c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:13:01.090565 10927 raft_consensus.cc:1261] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 1 FOLLOWER]: Refusing update from remote peer 3db16c7c878e4ea3a7b001c6dcec5c4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:13:01.090848 11125 consensus_queue.cc:1035] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b39bcb1f94e54d42b502442629cfad16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:13:01.090967 11131 consensus_queue.cc:1035] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aa3754841ae4e0e8ef163c9d6021cca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:13:01.093701 11136 mvcc.cc:205] Tried to move back new op lower bound from 7334449893744492544 to 7334449893707591680. Current Snapshot: MvccSnapshot[applied={T|T < 7334449893744492544}]
I20260928 23:13:01.093976 11135 mvcc.cc:205] Tried to move back new op lower bound from 7334449893744492544 to 7334449893707591680. Current Snapshot: MvccSnapshot[applied={T|T < 7334449893744492544 or (T in {7334449893744492544})}]
I20260928 23:13:01.095463 10647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:13:01.095553 10647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:13:01.096215 10647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6236a410145447a7830ea5b867021c87: Sending DeleteTablet for 3 replicas of tablet 5923b6e34b15420888b60fbd3342d6f9
I20260928 23:13:01.096534 11041 tablet_service.cc:1514] Processing DeleteTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44872
I20260928 23:13:01.096518 10907 tablet_service.cc:1514] Processing DeleteTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45512
I20260928 23:13:01.096791 11144 tablet_replica.cc:330] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: stopping tablet replica
I20260928 23:13:01.096812 10773 tablet_service.cc:1514] Processing DeleteTablet for tablet 5923b6e34b15420888b60fbd3342d6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41006
I20260928 23:13:01.096899 11144 raft_consensus.cc:2229] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:13:01.096983 11144 raft_consensus.cc:2258] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:13:01.096979 11146 tablet_replica.cc:330] T 5923b6e34b15420888b60fbd3342d6f9 P b39bcb1f94e54d42b502442629cfad16: stopping tablet replica
I20260928 23:13:01.097103 11145 tablet_replica.cc:330] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca: stopping tablet replica
I20260928 23:13:01.097201 11145 raft_consensus.cc:2229] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:13:01.097283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10710
I20260928 23:13:01.101553 11144 ts_tablet_manager.cc:1905] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:13:01.101835 11145 raft_consensus.cc:2258] T 5923b6e34b15420888b60fbd3342d6f9 P 4aa3754841ae4e0e8ef163c9d6021cca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:13:01.102363 10632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35021 (error 108)
I20260928 23:13:01.102669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10844
I20260928 23:13:01.103415 11144 ts_tablet_manager.cc:1918] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:13:01.103514 11144 log.cc:1196] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/5923b6e34b15420888b60fbd3342d6f9
I20260928 23:13:01.103775 11144 ts_tablet_manager.cc:1939] T 5923b6e34b15420888b60fbd3342d6f9 P 3db16c7c878e4ea3a7b001c6dcec5c4d: Deleting consensus metadata
I20260928 23:13:01.104138 10632 catalog_manager.cc:4841] TS 3db16c7c878e4ea3a7b001c6dcec5c4d (127.0.98.3:41955): tablet 5923b6e34b15420888b60fbd3342d6f9 (table test-table [id=af8909f8a1024db0a6df18a90f294fb8]) successfully deleted
I20260928 23:13:01.104655 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.104526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35021 (local address 127.0.0.1:41020)
0928 23:13:01.104581 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.104621 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 23:13:01.104797 10632 catalog_manager.cc:4584] TS b39bcb1f94e54d42b502442629cfad16 (127.0.98.1:35021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5923b6e34b15420888b60fbd3342d6f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35021: connect: Connection refused (error 111)
I20260928 23:13:01.107528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10978
I20260928 23:13:01.107558 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:01.107451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46221 (local address 127.0.0.1:45526)
0928 23:13:01.107498 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:01.107525 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:13:01.111617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 10616
2026-09-28T23:13:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1727 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:13:01 dist-test-slave-6vgd krb5kdc[11150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:13:01 dist-test-slave-6vgd krb5kdc[11150](info): set up 2 sockets
Sep 28 23:13:01 dist-test-slave-6vgd krb5kdc[11150](info): commencing operation
krb5kdc: starting...
W20260928 23:13:02.057129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:13:02 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:13:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 23:13:02.200749 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskawqjjE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/postgres -l logfile start
2026-09-28 23:13:04.456 UTC [11180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:13:04.456 UTC [11180] LOG: listening on IPv4 address "127.0.98.20", port 56423
2026-09-28 23:13:04.460 UTC [11180] LOG: listening on Unix socket "/tmp/.s.PGSQL.56423"
2026-09-28 23:13:04.485 UTC [11183] LOG: database system was shut down at 2026-09-28 23:13:03 UTC
2026-09-28 23:13:04.488 UTC [11180] LOG: database system is ready to accept connections
I20260928 23:13:05.352991 392 mini_postgres.cc:96] Postgres bound to 56423
2026-09-28 23:13:05.358 UTC [11191] FATAL: database "slave" does not exist
127.0.98.20:56423 - accepting connections
I20260928 23:13:05.359004 392 mini_ranger.cc:161] Starting Ranger...
I20260928 23:13:05.372172 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:13:05.791114 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:13:05.791185 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin
2026-09-28 23:13:06,062 [I] DB FLAVOR :POSTGRES
2026-09-28 23:13:06,063 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:13:06,063 [I] Checking connection..
2026-09-28 23:13:06,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:13:06,316 [I] Checking connection passed.
2026-09-28 23:13:06,316 [I] --------- Verifying version history table ---------
2026-09-28 23:13:06,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:06,550 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:13:06,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:06,777 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:13:06,778 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:13:06,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:13:07.015 UTC [11255] WARNING: there is no transaction in progress
2026-09-28 23:13:07,020 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:13:07,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:07,272 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:13:07,272 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:13:07,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:13:07,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:13:07,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6vgd.c.gcp-upstream.internal','N') ;"
2026-09-28 23:13:07,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:13:08,222 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:13:08,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:13:08,459 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:13:08,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:13:08,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:13:08,946 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:13:08,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:13:09.804 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.820 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.835 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.851 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.865 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.882 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.943 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.949 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.961 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.971 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.980 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:09.992 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:10.001 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:10.006 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:10.012 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:10.018 UTC [11363] WARNING: there is no transaction in progress
2026-09-28 23:13:10,559 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:13:10,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:13:10,799 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:13:10,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:13:11,037 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:13:11,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:13:11,260 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:13:11,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:13:11,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:13:11,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6vgd.c.gcp-upstream.internal';"
2026-09-28 23:13:11,946 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:13:11,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:13:12,193 [I] DB_PATCHES have already been applied
I20260928 23:13:12.198793 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskawqjjE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:13:12.198850 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 23:13:12.201471 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39409
Sep 28, 2026 11:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:13:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:13:14.228176 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.869s user 0.000s sys 0.009s
Sep 28, 2026 11:13:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:13:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 11:13:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:13:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:13:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:13:34.876423 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:13:34 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:13:34.970851 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:13:34.979506 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:13:35.781615 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:13:35.781700 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms
I20260928 23:13:35.781716 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:56423
2026-09-28 23:13:35,936 [I] DB FLAVOR :POSTGRES
2026-09-28 23:13:35,936 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:13:35,936 [I] Checking connection
2026-09-28 23:13:35,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:13:36,183 [I] connection success
2026-09-28 23:13:36,184 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:13:36,184 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:13:36,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:13:36,416 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:13:36,416 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:13:36,416 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:13:36,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:56423/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:13:36,689 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:13:37.142153 392 mini_ranger_kms.cc:310] Created kms service
I20260928 23:13:37.329875 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:13:37.371719 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:13:37 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637214, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:13:37.734687 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:13:37.734799 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:13:37.734827 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 23:13:37.738248 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11544
I20260928 23:13:37.738335 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34891
Sep 28, 2026 11:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:13:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:13:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 23:13:39.754593 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.784s user 0.000s sys 0.138s
I20260928 23:13:39.754711 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:13:39.754798 392 mini_ranger_kms.cc:401] 127.0.98.20:34891/kms/v1/keys
Sep 28 23:13:39 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 23:13:40 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:13:40 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:13:42 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:13:42.919852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37391
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34891/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:42.998644 11595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:42.998843 11595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:42.998863 11595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:43.000042 11595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:13:43.000083 11595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:43.000097 11595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:43.000108 11595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:13:43.000118 11595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:13:43.001658 11595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37391
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:43.001870 11595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:43.002092 11595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:43.006742 11603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:43.006798 11600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:43.006747 11601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:43.007037 11595 server_base.cc:1034] running on GCE node
I20260928 23:13:43.007249 11595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:43.007526 11595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:43.008675 11595 hybrid_clock.cc:648] HybridClock initialized: now 1790637223008664 us; error 26 us; skew 500 ppm
Sep 28 23:13:43 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:43.015982 11595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:13:43.016440 11595 webserver.cc:460] Webserver started at http://127.0.98.62:40347/ using document root <none> and password file <none>
I20260928 23:13:43.016649 11595 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:43.016688 11595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:43.016808 11595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:13:43 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:13:43.076454 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "3752fa4ec40745619606168d269ca155"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "f4e79a6bd6d4c2398e88c93c4ef5adf8"
server_key_iv: "517b3f4c633be372fbb9e6430786154e"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:43.076792 11595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "3752fa4ec40745619606168d269ca155"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "f4e79a6bd6d4c2398e88c93c4ef5adf8"
server_key_iv: "517b3f4c633be372fbb9e6430786154e"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:43.078094 11595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:13:43.089552 11610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:43.089746 11595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:13:43.089843 11595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "3752fa4ec40745619606168d269ca155"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "f4e79a6bd6d4c2398e88c93c4ef5adf8"
server_key_iv: "517b3f4c633be372fbb9e6430786154e"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:43.089931 11595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:43.113543 11595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:43.115306 11595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:43.115437 11595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:43.120059 11595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37391
I20260928 23:13:43.120108 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37391 every 8 connection(s)
I20260928 23:13:43.120424 11595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:13:43.121016 11663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:43.123165 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: Bootstrap starting.
I20260928 23:13:43.123701 11663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:43.123932 11663 log.cc:824] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:43.124521 11663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: No bootstrap required, opened a new log
I20260928 23:13:43.125268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 11595
I20260928 23:13:43.125375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:13:43.125562 392 external_mini_cluster.cc:1428] Setting key decdb041fcfee813a4a2e31664df87d2
I20260928 23:13:43.126893 11663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260928 23:13:43.127117 11663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:43.127188 11663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3752fa4ec40745619606168d269ca155, State: Initialized, Role: FOLLOWER
I20260928 23:13:43.127318 11663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260928 23:13:43.127388 11663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:13:43.127417 11663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:13:43.127475 11663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:43.128059 11663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260928 23:13:43.128161 11663 leader_election.cc:302] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3752fa4ec40745619606168d269ca155; no voters:
I20260928 23:13:43.128347 11663 leader_election.cc:288] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:13:43.128392 11668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:13:43.128608 11668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [term 1 LEADER]: Becoming Leader. State: Replica: 3752fa4ec40745619606168d269ca155, State: Running, Role: LEADER
I20260928 23:13:43.128655 11663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:13:43.128775 11668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } }
I20260928 23:13:43.129293 11670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3752fa4ec40745619606168d269ca155. Latest consensus state: current_term: 1 leader_uuid: "3752fa4ec40745619606168d269ca155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } } }
I20260928 23:13:43.129402 11670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [sys.catalog]: This master's current role is: LEADER
I20260928 23:13:43.129590 11677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:13:43.130060 11677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:13:43.130302 11663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:13:43.131098 11668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3752fa4ec40745619606168d269ca155" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3752fa4ec40745619606168d269ca155" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37391 } } }
I20260928 23:13:43.131237 11668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155 [sys.catalog]: This master's current role is: LEADER
Sep 28 23:13:43 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:43.137144 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:43.126037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53270 (local address 127.0.98.62:37391)
0928 23:13:43.126202 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:43.126212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:43.126243 (+ 31us) server_negotiation.cc:407] Connection header received
0928 23:13:43.126867 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:43.126875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:43.126936 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:43.127079 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:43.127789 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:43.129104 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:43.130258 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:43.130886 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:43.130991 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:43.133734 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:43.133738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:43.133739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:43.133751 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:43.135570 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:43.135935 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:43.135939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:43.135939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:43.136052 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:43.136250 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:43.136251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:43.136251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:43.136549 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:43.136692 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:43.136814 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:13:43.136935 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260928 23:13:43.141462 11677 catalog_manager.cc:1347] Generated new cluster ID: 45f86fe90edb4ae586ecae2b2b57ed19
I20260928 23:13:43.141577 11677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:13:43.149156 11677 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:13:43.149955 11677 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:13:43.155047 11677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: Generated new TSK 0
I20260928 23:13:43.155306 11677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:13:43.848115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:43.963503 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:43.963713 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:43.963734 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:43.964866 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:43.964907 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:43.964931 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:13:43.966382 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:43.966593 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:43.966806 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:43.971514 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:43.971524 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:43.971694 11709 server_base.cc:1034] running on GCE node
W20260928 23:13:43.971794 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:43.972112 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:43.972353 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:43.973497 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790637223973487 us; error 26 us; skew 500 ppm
Sep 28 23:13:43 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:43.976018 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:13:43.976524 11709 webserver.cc:460] Webserver started at http://127.0.98.1:40341/ using document root <none> and password file <none>
I20260928 23:13:43.976758 11709 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:43.976816 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:43.976984 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:13:43 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:13:43.991341 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "fab32a77b5c54e4d901d08501f284166"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "da789247e1b6d685fa665b66bb0dd017"
server_key_iv: "2b6d3327f1efa2bf39fff58bc83a99aa"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:43.991793 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fab32a77b5c54e4d901d08501f284166"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "da789247e1b6d685fa665b66bb0dd017"
server_key_iv: "2b6d3327f1efa2bf39fff58bc83a99aa"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:43.993690 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 23:13:44.013150 11724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.013464 11709 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260928 23:13:44.013582 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "fab32a77b5c54e4d901d08501f284166"
format_stamp: "Formatted at 2026-09-28 23:13:43 on dist-test-slave-6vgd"
server_key: "da789247e1b6d685fa665b66bb0dd017"
server_key_iv: "2b6d3327f1efa2bf39fff58bc83a99aa"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.013687 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:44.032346 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:44.034809 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:44.034958 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:44.035188 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:44.035518 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:44.035571 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.035604 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:44.035619 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.043246 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45175
I20260928 23:13:44.043308 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45175 every 8 connection(s)
I20260928 23:13:44.043701 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:13:44.044245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 11709
I20260928 23:13:44.044341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:13:44.044555 392 external_mini_cluster.cc:1428] Setting key f052b86dcb9cfcafd04c714c9127fa3d
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:13:44.054109 11842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.044414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.1:39843)
0928 23:13:44.044568 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.044612 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.045101 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.045488 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.045493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.045687 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:44.045893 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.045898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.046595 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.046596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.047792 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.047795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.048251 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.048252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.048381 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.048955 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:44.048966 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:44.050600 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:44.052536 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.052542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.052545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.052834 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.053052 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.053052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.053053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.053180 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.053495 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:44.053500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:44.053676 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.053784 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.053932 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 23:13:44.054570 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.044435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39843 (local address 127.0.98.62:37391)
0928 23:13:44.045299 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.045301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.045308 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:13:44.045327 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.045328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.045337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.045391 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:44.045977 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.046491 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.047887 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.048160 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.049611 (+ 1451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.050707 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:44.050709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:44.050710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:44.050717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:44.052443 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.052921 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.052922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.052923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.052979 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.053296 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.053296 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.053297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.053429 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:44.054356 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.054397 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.054475 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":84,"threads_started":1}
I20260928 23:13:44.054927 11840 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260928 23:13:44.055033 11840 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:44.055212 11840 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
I20260928 23:13:44.055745 11627 ts_manager.cc:194] Registered new tserver with Master: fab32a77b5c54e4d901d08501f284166 (127.0.98.1:45175)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 23:13:44.056454 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:13:44.070144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:44.176450 11846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:44.176731 11846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:44.176815 11846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:44.178622 11846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:44.178680 11846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:44.178717 11846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:13:44.180905 11846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:44.181154 11846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:44.181495 11846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:44.187628 11853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:44.187634 11854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:44.187929 11856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:44.188697 11846 server_base.cc:1034] running on GCE node
I20260928 23:13:44.188896 11846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:44.189188 11846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:44.190428 11846 hybrid_clock.cc:648] HybridClock initialized: now 1790637224190408 us; error 32 us; skew 500 ppm
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:44.193689 11846 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:13:44.194271 11846 webserver.cc:460] Webserver started at http://127.0.98.2:41937/ using document root <none> and password file <none>
I20260928 23:13:44.194543 11846 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:44.194612 11846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:44.194792 11846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:44.209993 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "c5db667f9bf34643ac1725aeb217785c"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "ab860850ea0f68579e47a36e4d140057"
server_key_iv: "31e53023da2490d59aa51b25dbe1ee08"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.210491 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c5db667f9bf34643ac1725aeb217785c"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "ab860850ea0f68579e47a36e4d140057"
server_key_iv: "31e53023da2490d59aa51b25dbe1ee08"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.228672 11846 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.001s sys 0.000s
I20260928 23:13:44.253333 11863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.253626 11846 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260928 23:13:44.253751 11846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "c5db667f9bf34643ac1725aeb217785c"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "ab860850ea0f68579e47a36e4d140057"
server_key_iv: "31e53023da2490d59aa51b25dbe1ee08"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.253854 11846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:44.264859 11846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:44.267181 11846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:44.267385 11846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:44.267622 11846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:44.268031 11846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:44.268124 11846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.268208 11846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:44.268261 11846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.274816 11846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40109
I20260928 23:13:44.275328 11846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:13:44.282248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 11846
I20260928 23:13:44.282348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:13:44.282526 392 external_mini_cluster.cc:1428] Setting key 81ac227ac025427db46d8944673e2a7d
I20260928 23:13:44.277948 11976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40109 every 8 connection(s)
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:44.292455 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.278815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59919 (local address 127.0.98.62:37391)
0928 23:13:44.278876 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.278878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.279141 (+ 263us) server_negotiation.cc:407] Connection header received
0928 23:13:44.280313 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.280314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.280327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.280355 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:44.281132 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.282016 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.286220 (+ 4204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.286500 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.286546 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.289179 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:44.289180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:44.289180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:44.289189 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:44.290815 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.291284 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.291286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.291286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.291340 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.291629 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.291630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.291631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.291786 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:44.291830 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.292266 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.292359 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:13:44.292999 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.278529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.2:59919)
0928 23:13:44.279019 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.279062 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.280190 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.280529 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.280533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.280760 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:44.281026 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.281031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.284716 (+ 3685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.284717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.286104 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.286107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.286808 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.286809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.286947 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.287498 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:44.287508 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:44.289064 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:44.290912 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.290918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.290921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.291195 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.291407 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.291408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.291408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.291549 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.291877 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:44.291883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:44.292066 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.292545 (+ 479us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.292716 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":85,"threads_started":1}
I20260928 23:13:44.293608 11977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260928 23:13:44.293705 11977 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:44.293900 11977 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
I20260928 23:13:44.294287 11627 ts_manager.cc:194] Registered new tserver with Master: c5db667f9bf34643ac1725aeb217785c (127.0.98.2:40109)
I20260928 23:13:44.294643 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:13:44.310410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:13:44.401319 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:13:44.401535 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:13:44.401567 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:13:44.402733 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:13:44.402781 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:13:44.402814 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:13:44.404309 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:13:44.404551 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:13:44.404799 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:13:44.409657 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:44.409657 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:13:44.410086 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:13:44.410367 11983 server_base.cc:1034] running on GCE node
I20260928 23:13:44.410557 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:13:44.410794 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:13:44.411936 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790637224411924 us; error 26 us; skew 500 ppm
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:13:44.414381 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:13:44.414852 11983 webserver.cc:460] Webserver started at http://127.0.98.3:37099/ using document root <none> and password file <none>
I20260928 23:13:44.415064 11983 fs_manager.cc:359] Metadata directory not provided
I20260928 23:13:44.415109 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:13:44.415237 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:13:44.435572 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "56658bbbf5894ddc9e06f568011229e6"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "6a8f1265c4ef85b578b8cc1dd2a1a64b"
server_key_iv: "7d7b7e8cb4eb32f2f1f0c794f5631e0c"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.436149 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56658bbbf5894ddc9e06f568011229e6"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "6a8f1265c4ef85b578b8cc1dd2a1a64b"
server_key_iv: "7d7b7e8cb4eb32f2f1f0c794f5631e0c"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.441682 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260928 23:13:44.458081 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.458355 11983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:13:44.458483 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "56658bbbf5894ddc9e06f568011229e6"
format_stamp: "Formatted at 2026-09-28 23:13:44 on dist-test-slave-6vgd"
server_key: "6a8f1265c4ef85b578b8cc1dd2a1a64b"
server_key_iv: "7d7b7e8cb4eb32f2f1f0c794f5631e0c"
server_key_version: "kuduclusterkey@0"
I20260928 23:13:44.458595 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:13:44.469980 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:13:44.472644 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:13:44.472803 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:13:44.473062 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:13:44.473491 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:13:44.473544 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.473578 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:13:44.473600 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:13:44.480001 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46319
I20260928 23:13:44.480465 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:13:44.483628 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46319 every 8 connection(s)
I20260928 23:13:44.485755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 11983
I20260928 23:13:44.485857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:13:44.486038 392 external_mini_cluster.cc:1428] Setting key 40a5384feec5af9f5292e637f88b8c61
Sep 28 23:13:44 dist-test-slave-6vgd krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:13:44.493451 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.484298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46509 (local address 127.0.98.62:37391)
0928 23:13:44.484347 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.484349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.484822 (+ 473us) server_negotiation.cc:407] Connection header received
0928 23:13:44.485477 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.485478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.485487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.485576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:44.486338 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.486838 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.488241 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.488430 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.488484 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.490685 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:44.490686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:44.490686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:44.490693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:44.491898 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.492336 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.492337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.492337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.492401 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.492646 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.492647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.492647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.492775 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:44.492826 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.493298 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.493385 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:13:44.493669 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.484257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37391 (local address 127.0.98.3:46509)
0928 23:13:44.484688 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.484732 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.485375 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.485673 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.485678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.485909 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:13:44.486220 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.486226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.486930 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.486931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.488158 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.488160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.488507 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.488508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.488645 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.489158 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:13:44.489168 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:13:44.490599 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:13:44.491972 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.491979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.491982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.492277 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.492452 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:13:44.492452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:13:44.492453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:13:44.492590 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:13:44.492847 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:13:44.492856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:13:44.493134 (+ 278us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.493274 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.493421 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":88,"threads_started":1}
I20260928 23:13:44.494153 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37391
I20260928 23:13:44.494254 12112 heartbeater.cc:464] Registering TS with master...
I20260928 23:13:44.494441 12112 heartbeater.cc:511] Master 127.0.98.62:37391 requested a full tablet report, sending...
I20260928 23:13:44.494794 11627 ts_manager.cc:194] Registered new tserver with Master: 56658bbbf5894ddc9e06f568011229e6 (127.0.98.3:46319)
I20260928 23:13:44.495147 11627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46509
I20260928 23:13:44.498098 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:13:44.505883 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.499436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.0.98.62:37391)
0928 23:13:44.499471 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.499473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.499603 (+ 130us) server_negotiation.cc:407] Connection header received
0928 23:13:44.499629 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.499630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.499641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.499702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:13:44.500190 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.500903 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.501768 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.502034 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.502094 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.503181 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:13:44.503183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:13:44.503183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:13:44.503191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:13:44.504700 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.505028 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.505029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.505030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.505107 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:13:44.505335 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:13:44.505336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:13:44.505336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:13:44.505464 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:13:44.505563 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.505705 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.505784 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:13:44.507560 11627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:13:44.509243 11627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:13:44.527155 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.511963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:54616)
0928 23:13:44.522602 (+ 10639us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.522614 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.522680 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.522950 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.522954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.522956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:44.523048 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.523051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.524319 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.524320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.525755 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.525758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.526828 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.526829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.526924 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.526937 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.527002 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.527065 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10619,"thread_start_us":67,"threads_started":1}
I20260928 23:13:44.527194 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.517830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.0.98.2:40109)
0928 23:13:44.522257 (+ 4427us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.522264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.523110 (+ 846us) server_negotiation.cc:407] Connection header received
0928 23:13:44.523237 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.523240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.523256 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.523306 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:44.523516 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.524526 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.525937 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.526699 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.526749 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.526821 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.527051 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.527091 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4379,"thread_start_us":102,"threads_started":1}
I20260928 23:13:44.527698 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.521287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.0.98.1:45175)
0928 23:13:44.522328 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.522332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.522678 (+ 346us) server_negotiation.cc:407] Connection header received
0928 23:13:44.522749 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.522752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.522777 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.522848 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:44.523131 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.524223 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.525879 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.526637 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.526705 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.526794 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.527576 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.527618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":73,"threads_started":1}
I20260928 23:13:44.528882 11774 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4554bd28034da18931f6a9f995968f (DEFAULT_TABLE table=test-table [id=5e2bbfd2b9b24627a3eb07eec01d6f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:44.529337 11774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4554bd28034da18931f6a9f995968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:44.532172 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.511758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46319 (local address 127.0.0.1:37056)
0928 23:13:44.522408 (+ 10650us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.522471 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.522552 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.522963 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.522964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.522966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:44.523050 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.523052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.528926 (+ 5874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.528927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.530126 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.530129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.531602 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.531603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.532000 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.532006 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.532036 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.532078 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10608,"thread_start_us":71,"threads_started":1}
I20260928 23:13:44.532517 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.512263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.0.98.3:46319)
0928 23:13:44.522811 (+ 10548us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.522815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.522823 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:13:44.522855 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.522857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.522869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.522937 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:44.527370 (+ 4433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.528817 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.530263 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.531503 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.532280 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.532372 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.532401 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.532441 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10509,"thread_start_us":95,"threads_started":1}
I20260928 23:13:44.533164 12046 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4554bd28034da18931f6a9f995968f (DEFAULT_TABLE table=test-table [id=5e2bbfd2b9b24627a3eb07eec01d6f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:44.533511 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4554bd28034da18931f6a9f995968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:44.528108 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.515481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40109 (local address 127.0.0.1:35484)
0928 23:13:44.523046 (+ 7565us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.523055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.523137 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.523411 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.523411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.523413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:44.523448 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.523449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.524660 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.524661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.525811 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.525814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.526882 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.526883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.526950 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.526954 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.527006 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.528022 (+ 1016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7549,"mutex_wait_us":965,"thread_start_us":81,"threads_started":1}
I20260928 23:13:44.534344 11911 tablet_service.cc:1467] Processing CreateTablet for tablet 2a4554bd28034da18931f6a9f995968f (DEFAULT_TABLE table=test-table [id=5e2bbfd2b9b24627a3eb07eec01d6f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:13:44.534670 11911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a4554bd28034da18931f6a9f995968f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:13:44.537788 12130 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Bootstrap starting.
I20260928 23:13:44.538652 12132 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Bootstrap starting.
I20260928 23:13:44.538830 12130 tablet_bootstrap.cc:654] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:44.539132 12130 log.cc:824] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:44.539482 12131 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Bootstrap starting.
I20260928 23:13:44.539740 12132 tablet_bootstrap.cc:654] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:44.540077 12132 log.cc:824] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:44.540347 12131 tablet_bootstrap.cc:654] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Neither blocks nor log segments found. Creating new log.
I20260928 23:13:44.540637 12131 log.cc:824] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:13:44.540854 12132 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: No bootstrap required, opened a new log
I20260928 23:13:44.540906 12132 ts_tablet_manager.cc:1397] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:13:44.541375 12130 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: No bootstrap required, opened a new log
I20260928 23:13:44.541430 12130 ts_tablet_manager.cc:1397] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:13:44.542799 12132 raft_consensus.cc:348] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.542958 12132 raft_consensus.cc:374] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:44.542992 12132 raft_consensus.cc:729] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5db667f9bf34643ac1725aeb217785c, State: Initialized, Role: FOLLOWER
I20260928 23:13:44.543103 12132 consensus_queue.cc:260] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.543342 12132 ts_tablet_manager.cc:1428] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:13:44.543411 12130 raft_consensus.cc:348] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.543545 11977 heartbeater.cc:503] Master 127.0.98.62:37391 was elected leader, sending a full tablet report...
I20260928 23:13:44.543568 12130 raft_consensus.cc:374] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:44.543602 12130 raft_consensus.cc:729] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab32a77b5c54e4d901d08501f284166, State: Initialized, Role: FOLLOWER
I20260928 23:13:44.543704 12130 consensus_queue.cc:260] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.543927 12130 ts_tablet_manager.cc:1428] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:13:44.544031 11840 heartbeater.cc:503] Master 127.0.98.62:37391 was elected leader, sending a full tablet report...
W20260928 23:13:44.544355 11841 tablet.cc:2369] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:13:44.545042 12131 tablet_bootstrap.cc:492] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: No bootstrap required, opened a new log
I20260928 23:13:44.545115 12131 ts_tablet_manager.cc:1397] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 23:13:44.547634 12131 raft_consensus.cc:348] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.547806 12131 raft_consensus.cc:374] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:13:44.547839 12131 raft_consensus.cc:729] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56658bbbf5894ddc9e06f568011229e6, State: Initialized, Role: FOLLOWER
I20260928 23:13:44.547976 12131 consensus_queue.cc:260] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.548259 12131 ts_tablet_manager.cc:1428] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:13:44.548401 12112 heartbeater.cc:503] Master 127.0.98.62:37391 was elected leader, sending a full tablet report...
I20260928 23:13:44.578483 12138 raft_consensus.cc:482] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:13:44.578649 12138 raft_consensus.cc:504] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.579041 12138 leader_election.cc:288] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5db667f9bf34643ac1725aeb217785c (127.0.98.2:40109), fab32a77b5c54e4d901d08501f284166 (127.0.98.1:45175)
I20260928 23:13:44.582672 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.579184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40109 (local address 127.0.98.3:56827)
0928 23:13:44.579280 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.579291 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.579334 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.579502 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.579503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.579504 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:44.579581 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.579583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.580485 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.580485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.581701 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.581703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.582429 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.582430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.582502 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.582506 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.582546 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.582584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3}
I20260928 23:13:44.582832 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.579264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57917 (local address 127.0.98.1:45175)
0928 23:13:44.579314 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.579316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.579738 (+ 422us) server_negotiation.cc:407] Connection header received
0928 23:13:44.579823 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.579825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.579835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.579863 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:44.580056 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.580699 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.581957 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.582503 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.582540 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.582548 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.582728 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.582769 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:13:44.582933 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.579359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56827 (local address 127.0.98.2:40109)
0928 23:13:44.579397 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.579399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.579406 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:13:44.579431 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.579431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.579441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.579484 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:13:44.579656 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.580384 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.581810 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.582331 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.582390 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.582401 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.582837 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.582874 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:13:44.583037 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.579184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.98.3:57917)
0928 23:13:44.579684 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.579693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:13:44.579759 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:13:44.579918 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:13:44.579919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:13:44.579932 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:13:44.579994 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.579996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.580777 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.580778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.581877 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:13:44.581878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.582595 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.582596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:13:44.582656 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.582659 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:13:44.582929 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 23:13:44.582965 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":335,"thread_start_us":87,"threads_started":1}
I20260928 23:13:44.583186 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4554bd28034da18931f6a9f995968f" candidate_uuid: "56658bbbf5894ddc9e06f568011229e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5db667f9bf34643ac1725aeb217785c" is_pre_election: true
I20260928 23:13:44.583353 11931 raft_consensus.cc:2454] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56658bbbf5894ddc9e06f568011229e6 in term 0.
I20260928 23:13:44.583313 11794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4554bd28034da18931f6a9f995968f" candidate_uuid: "56658bbbf5894ddc9e06f568011229e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab32a77b5c54e4d901d08501f284166" is_pre_election: true
I20260928 23:13:44.583468 11794 raft_consensus.cc:2454] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56658bbbf5894ddc9e06f568011229e6 in term 0.
I20260928 23:13:44.583580 11999 leader_election.cc:302] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56658bbbf5894ddc9e06f568011229e6, c5db667f9bf34643ac1725aeb217785c; no voters:
I20260928 23:13:44.583739 12138 raft_consensus.cc:2781] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:13:44.583807 12138 raft_consensus.cc:482] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:13:44.583837 12138 raft_consensus.cc:3037] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:44.584499 12138 raft_consensus.cc:504] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.584615 12138 leader_election.cc:288] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [CANDIDATE]: Term 1 election: Requested vote from peers c5db667f9bf34643ac1725aeb217785c (127.0.98.2:40109), fab32a77b5c54e4d901d08501f284166 (127.0.98.1:45175)
I20260928 23:13:44.584789 11794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4554bd28034da18931f6a9f995968f" candidate_uuid: "56658bbbf5894ddc9e06f568011229e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab32a77b5c54e4d901d08501f284166"
I20260928 23:13:44.584843 11931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a4554bd28034da18931f6a9f995968f" candidate_uuid: "56658bbbf5894ddc9e06f568011229e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5db667f9bf34643ac1725aeb217785c"
I20260928 23:13:44.584867 11794 raft_consensus.cc:3037] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:44.584903 11931 raft_consensus.cc:3037] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:13:44.585709 11931 raft_consensus.cc:2454] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56658bbbf5894ddc9e06f568011229e6 in term 1.
I20260928 23:13:44.585708 11794 raft_consensus.cc:2454] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56658bbbf5894ddc9e06f568011229e6 in term 1.
I20260928 23:13:44.585870 11999 leader_election.cc:302] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56658bbbf5894ddc9e06f568011229e6, c5db667f9bf34643ac1725aeb217785c; no voters:
I20260928 23:13:44.585953 12138 raft_consensus.cc:2781] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:13:44.586086 12138 raft_consensus.cc:686] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 1 LEADER]: Becoming Leader. State: Replica: 56658bbbf5894ddc9e06f568011229e6, State: Running, Role: LEADER
I20260928 23:13:44.586179 12138 consensus_queue.cc:237] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } }
I20260928 23:13:44.586937 11626 catalog_manager.cc:5495] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56658bbbf5894ddc9e06f568011229e6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "56658bbbf5894ddc9e06f568011229e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56658bbbf5894ddc9e06f568011229e6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 } health_report { overall_health: UNKNOWN } } }
I20260928 23:13:44.593387 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.591140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.98.3:46319)
0928 23:13:44.591175 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:13:44.591176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:13:44.591184 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:13:44.591255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:13:44.591256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:13:44.591279 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:13:44.591335 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:13:44.591431 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.591888 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.592632 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:13:44.592883 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:13:44.592938 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:13:44.593079 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:13:44.593179 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:13:44.593266 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:13:44.593300 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:13:44.593318 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:13:44.595422 11794 raft_consensus.cc:1261] T 2a4554bd28034da18931f6a9f995968f P fab32a77b5c54e4d901d08501f284166 [term 1 FOLLOWER]: Refusing update from remote peer 56658bbbf5894ddc9e06f568011229e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:13:44.595744 12138 consensus_queue.cc:1035] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fab32a77b5c54e4d901d08501f284166" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:13:44.597509 11931 raft_consensus.cc:1261] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 1 FOLLOWER]: Refusing update from remote peer 56658bbbf5894ddc9e06f568011229e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:13:44.597781 12140 consensus_queue.cc:1035] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5db667f9bf34643ac1725aeb217785c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:13:44.600422 12143 mvcc.cc:205] Tried to move back new op lower bound from 7334450071939325952 to 7334450071905554432. Current Snapshot: MvccSnapshot[applied={T|T < 7334450071939325952}]
I20260928 23:13:44.600834 12146 mvcc.cc:205] Tried to move back new op lower bound from 7334450071939325952 to 7334450071905554432. Current Snapshot: MvccSnapshot[applied={T|T < 7334450071939325952}]
I20260928 23:13:44.602065 12144 mvcc.cc:205] Tried to move back new op lower bound from 7334450071939325952 to 7334450071905554432. Current Snapshot: MvccSnapshot[applied={T|T < 7334450071939325952}]
I20260928 23:13:44.604270 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:13:44.604359 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:13:44.605057 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3752fa4ec40745619606168d269ca155: Sending DeleteTablet for 3 replicas of tablet 2a4554bd28034da18931f6a9f995968f
I20260928 23:13:44.605386 11774 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4554bd28034da18931f6a9f995968f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54616
I20260928 23:13:44.605407 11911 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4554bd28034da18931f6a9f995968f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35484
I20260928 23:13:44.605587 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a4554bd28034da18931f6a9f995968f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:13:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37056
I20260928 23:13:44.605926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 11709
I20260928 23:13:44.613304 12154 tablet_replica.cc:330] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: stopping tablet replica
I20260928 23:13:44.613420 12154 raft_consensus.cc:2229] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:13:44.613562 12154 raft_consensus.cc:2258] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:13:44.613742 12153 tablet_replica.cc:330] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: stopping tablet replica
I20260928 23:13:44.613852 12153 raft_consensus.cc:2229] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:13:44.613957 12153 raft_consensus.cc:2258] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:13:44.615959 12154 ts_tablet_manager.cc:1905] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:13:44.617786 12153 ts_tablet_manager.cc:1905] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:13:44.619139 12154 ts_tablet_manager.cc:1918] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:13:44.619204 12154 log.cc:1196] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/2a4554bd28034da18931f6a9f995968f
I20260928 23:13:44.619431 12154 ts_tablet_manager.cc:1939] T 2a4554bd28034da18931f6a9f995968f P 56658bbbf5894ddc9e06f568011229e6: Deleting consensus metadata
I20260928 23:13:44.619846 11612 catalog_manager.cc:4841] TS 56658bbbf5894ddc9e06f568011229e6 (127.0.98.3:46319): tablet 2a4554bd28034da18931f6a9f995968f (table test-table [id=5e2bbfd2b9b24627a3eb07eec01d6f55]) successfully deleted
I20260928 23:13:44.621042 12153 ts_tablet_manager.cc:1918] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:13:44.621114 12153 log.cc:1196] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/2a4554bd28034da18931f6a9f995968f
I20260928 23:13:44.621577 12153 ts_tablet_manager.cc:1939] T 2a4554bd28034da18931f6a9f995968f P c5db667f9bf34643ac1725aeb217785c: Deleting consensus metadata
I20260928 23:13:44.621959 11611 catalog_manager.cc:4841] TS c5db667f9bf34643ac1725aeb217785c (127.0.98.2:40109): tablet 2a4554bd28034da18931f6a9f995968f (table test-table [id=5e2bbfd2b9b24627a3eb07eec01d6f55]) successfully deleted
W20260928 23:13:44.627849 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45175 (error 108)
I20260928 23:13:44.628206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 11846
I20260928 23:13:44.634850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 11983
I20260928 23:13:44.637753 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:13:44.637628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45175 (local address 127.0.0.1:54628)
0928 23:13:44.637681 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:13:44.637715 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:13:44.637962 11612 catalog_manager.cc:4584] TS fab32a77b5c54e4d901d08501f284166 (127.0.98.1:45175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a4554bd28034da18931f6a9f995968f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45175: connect: Connection refused (error 111)
I20260928 23:13:44.641162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 11595
I20260928 23:13:44.646945 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:13:44.980551 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:13:44.980646 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:13:45.314177 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:13:45.314 UTC [11180] LOG: received smart shutdown request
2026-09-28 23:13:45.317 UTC [11180] LOG: background worker "logical replication launcher" (PID 11189) exited with exit code 1
2026-09-28 23:13:45.317 UTC [11184] LOG: shutting down
2026-09-28 23:13:45.453 UTC [11180] LOG: database system is shut down
2026-09-28T23:13:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44423 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:13:45 dist-test-slave-6vgd krb5kdc[12164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:13:45 dist-test-slave-6vgd krb5kdc[12164](info): set up 2 sockets
Sep 28 23:13:45 dist-test-slave-6vgd krb5kdc[12164](info): commencing operation
krb5kdc: starting...
W20260928 23:13:46.484844 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:13:46 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:13:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 23:13:46.633867 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskawqjjE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/postgres -l logfile start
2026-09-28 23:13:48.843 UTC [12194] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 23:13:48.843 UTC [12194] LOG: listening on IPv4 address "127.0.98.20", port 39059
2026-09-28 23:13:48.847 UTC [12194] LOG: listening on Unix socket "/tmp/.s.PGSQL.39059"
2026-09-28 23:13:48.871 UTC [12197] LOG: database system was shut down at 2026-09-28 23:13:47 UTC
2026-09-28 23:13:48.875 UTC [12194] LOG: database system is ready to accept connections
I20260928 23:13:49.732841 392 mini_postgres.cc:96] Postgres bound to 39059
2026-09-28 23:13:49.738 UTC [12205] FATAL: database "slave" does not exist
127.0.98.20:39059 - accepting connections
I20260928 23:13:49.738782 392 mini_ranger.cc:161] Starting Ranger...
I20260928 23:13:49.752594 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 23:13:50.179773 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 23:13:50.179858 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin
2026-09-28 23:13:50,430 [I] DB FLAVOR :POSTGRES
2026-09-28 23:13:50,431 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:13:50,431 [I] Checking connection..
2026-09-28 23:13:50,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 23:13:50,677 [I] Checking connection passed.
2026-09-28 23:13:50,677 [I] --------- Verifying version history table ---------
2026-09-28 23:13:50,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:50,904 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:13:50,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:51,149 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 23:13:51,149 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 23:13:51,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 23:13:51.396 UTC [12269] WARNING: there is no transaction in progress
2026-09-28 23:13:51,402 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 23:13:51,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 23:13:51,636 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 23:13:51,636 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:13:51,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 23:13:51,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 23:13:52,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6vgd.c.gcp-upstream.internal','N') ;"
2026-09-28 23:13:52,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:13:52,612 [I] Table x_portal_user does not exist in database ranger
2026-09-28 23:13:52,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:13:52,867 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 23:13:52,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:13:53,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:13:53,320 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 23:13:53,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 23:13:54.206 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.221 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.235 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.250 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.264 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.281 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.333 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.339 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.351 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.360 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.369 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.381 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.390 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.396 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.402 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54.407 UTC [12377] WARNING: there is no transaction in progress
2026-09-28 23:13:54,962 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 23:13:54,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 23:13:55,202 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 23:13:55,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 23:13:55,448 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 23:13:55,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 23:13:55,688 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 23:13:55,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 23:13:55,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 23:13:56,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6vgd.c.gcp-upstream.internal';"
2026-09-28 23:13:56,381 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 23:13:56,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 23:13:56,614 [I] DB_PATCHES have already been applied
I20260928 23:13:56.620039 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskawqjjE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 23:13:56.620102 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 23:13:56.622792 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56443
Sep 28, 2026 11:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:13:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:13:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 11:13:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 11:13:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 11:13:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 11:13:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 23:13:57.597882 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.859s user 0.004s sys 0.004s
Sep 28, 2026 11:14:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 11:14:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 11:14:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 11:14:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 11:14:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 23:14:17.431042 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 23:14:17 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 23:14:17.505242 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 23:14:17.513731 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 23:14:18.287532 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 23:14:18.287604 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms
I20260928 23:14:18.287621 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:39059
2026-09-28 23:14:18,431 [I] DB FLAVOR :POSTGRES
2026-09-28 23:14:18,431 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 23:14:18,431 [I] Checking connection
2026-09-28 23:14:18,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 23:14:18,660 [I] connection success
2026-09-28 23:14:18,661 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 23:14:18,661 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 23:14:18,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 23:14:18,885 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 23:14:18,885 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 23:14:18,885 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 23:14:18,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:39059/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 23:14:19,133 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 23:14:19.551281 392 mini_ranger_kms.cc:310] Created kms service
I20260928 23:14:19.726419 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 23:14:19.766634 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 23:14:19 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637257, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:20.138034 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 23:14:20.138260 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskawqjjE/build/release/bin/postgresql.jar:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskawqjjE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 23:14:20.138288 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 23:14:20.141319 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12557
I20260928 23:14:20.141392 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37557
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 11:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 23:14:22 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:22.168891 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.664s user 0.004s sys 0.123s
I20260928 23:14:22.168998 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 23:14:22.169042 392 mini_ranger_kms.cc:401] 127.0.98.20:37557/kms/v1/keys
Sep 28 23:14:22 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:14:22 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637262, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 23:14:25 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:14:25.324055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45897
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:25.402820 12608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:25.403031 12608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:25.403062 12608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:25.404153 12608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:25.404194 12608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:25.404213 12608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:25.404228 12608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:25.404245 12608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:25.405864 12608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37557/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45897
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:25.406095 12608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:25.406296 12608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:25.411366 12613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:25.411487 12608 server_base.cc:1034] running on GCE node
W20260928 23:14:25.411366 12614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:25.411397 12616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:25.411854 12608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:25.412156 12608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:25.413276 12608 hybrid_clock.cc:648] HybridClock initialized: now 1790637265413259 us; error 31 us; skew 500 ppm
Sep 28 23:14:25 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:25.420310 12608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:25.420770 12608 webserver.cc:460] Webserver started at http://127.0.98.62:35337/ using document root <none> and password file <none>
I20260928 23:14:25.420970 12608 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:25.421020 12608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:25.421133 12608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:14:25 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637265, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:25.479323 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "be440a6578254c67b4b9a5d971b90a18"
format_stamp: "Formatted at 2026-09-28 23:14:25 on dist-test-slave-6vgd"
server_key: "731ba58b322019712dfc99a1ca505a62"
server_key_iv: "b73c88f689064e7f184481d62e96249c"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:25.479713 12608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "be440a6578254c67b4b9a5d971b90a18"
format_stamp: "Formatted at 2026-09-28 23:14:25 on dist-test-slave-6vgd"
server_key: "731ba58b322019712dfc99a1ca505a62"
server_key_iv: "b73c88f689064e7f184481d62e96249c"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:25.481017 12608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:14:25.496520 12623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:25.496778 12608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:25.496879 12608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "be440a6578254c67b4b9a5d971b90a18"
format_stamp: "Formatted at 2026-09-28 23:14:25 on dist-test-slave-6vgd"
server_key: "731ba58b322019712dfc99a1ca505a62"
server_key_iv: "b73c88f689064e7f184481d62e96249c"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:25.496977 12608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:25.508605 12608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:25.510460 12608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:25.510589 12608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:25.514926 12608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45897
I20260928 23:14:25.514990 12675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45897 every 8 connection(s)
I20260928 23:14:25.515326 12608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:25.515954 12676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:25.518494 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Bootstrap starting.
I20260928 23:14:25.519042 12676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:25.519294 12676 log.cc:824] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:25.519606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 12608
I20260928 23:14:25.519696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:14:25.519869 392 external_mini_cluster.cc:1428] Setting key 59318fa1180a335b07d6b38be07a7048
I20260928 23:14:25.519999 12676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: No bootstrap required, opened a new log
I20260928 23:14:25.522863 12676 raft_consensus.cc:348] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:25.523037 12676 raft_consensus.cc:374] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:25.523069 12676 raft_consensus.cc:729] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be440a6578254c67b4b9a5d971b90a18, State: Initialized, Role: FOLLOWER
I20260928 23:14:25.523236 12676 consensus_queue.cc:260] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:25.523319 12676 raft_consensus.cc:388] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:25.523370 12676 raft_consensus.cc:482] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:25.523440 12676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:25.524091 12676 raft_consensus.cc:504] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:25.524199 12676 leader_election.cc:302] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be440a6578254c67b4b9a5d971b90a18; no voters:
I20260928 23:14:25.524389 12676 leader_election.cc:288] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:14:25.524595 12682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:25.524758 12682 raft_consensus.cc:686] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 LEADER]: Becoming Leader. State: Replica: be440a6578254c67b4b9a5d971b90a18, State: Running, Role: LEADER
I20260928 23:14:25.524766 12676 sys_catalog.cc:564] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:25.524892 12682 consensus_queue.cc:237] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:25.525246 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be440a6578254c67b4b9a5d971b90a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260928 23:14:25.525309 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:25.525418 12682 sys_catalog.cc:455] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be440a6578254c67b4b9a5d971b90a18. Latest consensus state: current_term: 1 leader_uuid: "be440a6578254c67b4b9a5d971b90a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260928 23:14:25.525460 12682 sys_catalog.cc:458] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:25.526202 12676 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 23:14:25 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:25.532886 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:25.520320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.0.98.62:45897)
0928 23:14:25.520502 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:25.520511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:25.520539 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:14:25.521362 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:25.521369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:25.521421 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:25.521543 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:25.522140 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:25.523007 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:25.523877 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:25.524331 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:25.524429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:25.528057 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:25.528116 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:25.528117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:25.528223 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:25.531103 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:25.531540 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:25.531543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:25.531543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:25.531649 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:25.531866 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:25.531867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:25.531867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:25.532209 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:25.532356 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:25.532496 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:14:25.532642 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260928 23:14:25.537247 12688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:25.537849 12688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:25.540031 12688 catalog_manager.cc:1347] Generated new cluster ID: 2971b055fa784474899b0e2b67b47f39
I20260928 23:14:25.540076 12688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:25.549038 12688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:14:25.549891 12688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:25.554865 12688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Generated new TSK 0
I20260928 23:14:25.555375 12688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:14:26.255317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:26.362251 12721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:26.362537 12721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:26.362573 12721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:26.364357 12721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:26.364421 12721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:26.364470 12721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:26.366715 12721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:26.366977 12721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:26.367267 12721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:26.373525 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:26.373642 12721 server_base.cc:1034] running on GCE node
W20260928 23:14:26.373525 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:26.373771 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:26.373956 12721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:26.374174 12721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:26.375310 12721 hybrid_clock.cc:648] HybridClock initialized: now 1790637266375297 us; error 30 us; skew 500 ppm
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:26.378019 12721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:26.378571 12721 webserver.cc:460] Webserver started at http://127.0.98.1:41267/ using document root <none> and password file <none>
I20260928 23:14:26.378829 12721 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:26.378902 12721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:26.379040 12721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:26.392678 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "927290f7fd8a4a8da2bfb794caad74c5"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "1be44fb9910c3481640b1d6486a32dc7"
server_key_iv: "1d3eb8f7d99d8fd0bf3ae1fe1ea8e237"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.393198 12721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "927290f7fd8a4a8da2bfb794caad74c5"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "1be44fb9910c3481640b1d6486a32dc7"
server_key_iv: "1d3eb8f7d99d8fd0bf3ae1fe1ea8e237"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.395449 12721 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 23:14:26.413357 12738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.413591 12721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 23:14:26.413686 12721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "927290f7fd8a4a8da2bfb794caad74c5"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "1be44fb9910c3481640b1d6486a32dc7"
server_key_iv: "1d3eb8f7d99d8fd0bf3ae1fe1ea8e237"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.413784 12721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:26.431198 12721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:26.433748 12721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:26.433960 12721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:26.434228 12721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:26.434648 12721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:26.434731 12721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.434790 12721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:26.434835 12721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.441136 12721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46117
I20260928 23:14:26.441648 12721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:14:26.442082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 12721
I20260928 23:14:26.442174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:14:26.442375 392 external_mini_cluster.cc:1428] Setting key 31ce6593bb261eab4e21374eac8907ed
I20260928 23:14:26.451364 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46117 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:26.466162 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.452284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.1:33673)
0928 23:14:26.453571 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:26.453626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:26.454417 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:26.454863 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:26.454869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:26.455195 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:26.455477 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.455484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.458330 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.458331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.459633 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.459636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.460175 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.460176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.460319 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.460942 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:26.460953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:26.462562 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:26.464468 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.464475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.464478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.464793 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.464990 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.464991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.464991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.465120 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.465446 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:26.465452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:26.465650 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 23:14:26.465759 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:14:26.465919 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":90,"threads_started":1}
I20260928 23:14:26.466496 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.452299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33673 (local address 127.0.98.62:45897)
0928 23:14:26.454622 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:26.454624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:26.454632 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:26.454656 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:26.454656 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:26.454669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:26.454727 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:26.457404 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.458217 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.459728 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.460032 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.460094 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.462663 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:26.462664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:26.462665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:26.462672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:26.464379 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.464872 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.464873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.464874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.464931 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.465187 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.465188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.465188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.465359 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:26.465420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:26.466311 (+ 891us) server_negotiation.cc:299] Negotiation successful
0928 23:14:26.466396 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292,"thread_start_us":95,"threads_started":1}
I20260928 23:14:26.468716 12852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:26.468824 12852 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:26.469039 12852 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:26.469769 12640 ts_manager.cc:194] Registered new tserver with Master: 927290f7fd8a4a8da2bfb794caad74c5 (127.0.98.1:46117)
I20260928 23:14:26.470618 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:14:26.498603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 23:14:26.615767 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:26.615974 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:26.616006 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:26.617158 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:26.617198 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:26.617259 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:26.619361 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:26.619807 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:26.620150 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:26.628695 12868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:26.628998 12865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:26.629976 12860 server_base.cc:1034] running on GCE node
W20260928 23:14:26.630399 12866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:26.630663 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:26.630995 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:26.632169 12860 hybrid_clock.cc:648] HybridClock initialized: now 1790637266632141 us; error 42 us; skew 500 ppm
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:26.635253 12860 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:26.635738 12860 webserver.cc:460] Webserver started at http://127.0.98.2:34449/ using document root <none> and password file <none>
I20260928 23:14:26.635964 12860 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:26.636015 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:26.636143 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:26.649111 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c051bc6fecd4fd0844d983e88242d4a"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "3c36c119435813d284016d675896298e"
server_key_iv: "d774d8212c07011eed23043acb132cee"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.649608 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c051bc6fecd4fd0844d983e88242d4a"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "3c36c119435813d284016d675896298e"
server_key_iv: "d774d8212c07011eed23043acb132cee"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.651053 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:14:26.681392 12876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.681602 12860 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 23:14:26.681696 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "0c051bc6fecd4fd0844d983e88242d4a"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "3c36c119435813d284016d675896298e"
server_key_iv: "d774d8212c07011eed23043acb132cee"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.681804 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:26.695859 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:26.698228 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:26.698379 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:26.698606 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:26.698971 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:26.699014 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.699047 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:26.699064 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.705251 12860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37347
I20260928 23:14:26.705668 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:26.715179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 12860
I20260928 23:14:26.715286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:14:26.715504 392 external_mini_cluster.cc:1428] Setting key 161ceb33697239f8ae2b474d72bc03a4
I20260928 23:14:26.718253 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37347 every 8 connection(s)
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:26.734505 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.718625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57095 (local address 127.0.98.62:45897)
0928 23:14:26.718679 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:26.718682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:26.721512 (+ 2830us) server_negotiation.cc:407] Connection header received
0928 23:14:26.722238 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:26.722240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:26.722252 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:26.722279 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:26.723119 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.723941 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.725470 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.725790 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.725830 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.730636 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:26.730638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:26.730638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:26.730646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:26.732290 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.732794 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.732795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.732796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.732858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.733689 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.733690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.733690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.733843 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:26.733914 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:26.734314 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 23:14:26.734402 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:14:26.735044 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.718645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.2:57095)
0928 23:14:26.721339 (+ 2694us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:26.721424 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:26.722112 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:26.722484 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:26.722489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:26.722739 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:26.723011 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.723017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.724052 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.724053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.725366 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.725370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.725925 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.725926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.726063 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.728809 (+ 2746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:26.728820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:26.730524 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:26.732400 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.732407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.732410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.732703 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.732927 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.732928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.732928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.733064 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.733951 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:26.733959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:26.734150 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 23:14:26.734607 (+ 457us) client_negotiation.cc:240] Negotiation successful
0928 23:14:26.734773 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2640,"thread_start_us":87,"threads_started":1}
I20260928 23:14:26.735505 12990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:26.735608 12990 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:26.735802 12990 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:26.736157 12640 ts_manager.cc:194] Registered new tserver with Master: 0c051bc6fecd4fd0844d983e88242d4a (127.0.98.2:37347)
I20260928 23:14:26.736644 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57095
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:14:26.765616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:26.890138 12995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:26.890476 12995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:26.890554 12995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:26.892246 12995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:26.892307 12995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:26.892345 12995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:26.894399 12995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:26.894856 12995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:26.895166 12995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:26.901525 13001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:26.901926 13000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:26.902150 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:26.903070 12995 server_base.cc:1034] running on GCE node
I20260928 23:14:26.903365 12995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:26.903644 12995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:26.904785 12995 hybrid_clock.cc:648] HybridClock initialized: now 1790637266904764 us; error 31 us; skew 500 ppm
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:26.908046 12995 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:26.908571 12995 webserver.cc:460] Webserver started at http://127.0.98.3:33087/ using document root <none> and password file <none>
I20260928 23:14:26.908823 12995 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:26.908871 12995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:26.908994 12995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:26.922338 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "ea6dfda1c1e9435b55c57c0178b27681"
server_key_iv: "f755eb76a2b182b1a29acce9ab60b4f1"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.922777 12995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "ea6dfda1c1e9435b55c57c0178b27681"
server_key_iv: "f755eb76a2b182b1a29acce9ab60b4f1"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.924535 12995 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:26.940167 13010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.940333 12995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:26.940430 12995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "ea6dfda1c1e9435b55c57c0178b27681"
server_key_iv: "f755eb76a2b182b1a29acce9ab60b4f1"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:26.940518 12995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:26.948930 12995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:26.951251 12995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:26.951402 12995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:26.951660 12995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:26.952024 12995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:26.952075 12995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.952108 12995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:26.952129 12995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:26.959167 12995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38299
I20260928 23:14:26.959604 12995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:14:26.962196 13123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38299 every 8 connection(s)
I20260928 23:14:26.963301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 12995
I20260928 23:14:26.963400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:14:26.963634 392 external_mini_cluster.cc:1428] Setting key c047d78bebc369717fef562b52985cab
Sep 28 23:14:26 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:26.974721 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.963099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54929 (local address 127.0.98.62:45897)
0928 23:14:26.963147 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:26.963149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:26.963570 (+ 421us) server_negotiation.cc:407] Connection header received
0928 23:14:26.965009 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:26.965010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:26.965023 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:26.965051 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:26.965913 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.966704 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.968220 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.968526 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.968572 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.971360 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:26.971361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:26.971362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:26.971369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:26.972981 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.973506 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.973507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.973508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.973571 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:26.973842 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:26.973843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:26.973844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:26.973980 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:26.974030 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:26.974519 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 23:14:26.974633 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:14:26.976035 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:26.962819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.3:54929)
0928 23:14:26.963396 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:26.963463 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:26.964885 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:26.965264 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:26.965268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:26.965529 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:26.965807 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.965813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.966811 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.966812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.968090 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:26.968093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:26.968673 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:26.968674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:26.968809 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:26.969391 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:26.969401 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:26.971250 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:26.973071 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.973077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.973079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.973399 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.973643 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:26.973644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:26.973644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:26.973769 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:26.974064 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:26.974070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:26.974282 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 23:14:26.975637 (+ 1355us) client_negotiation.cc:240] Negotiation successful
0928 23:14:26.975804 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":79,"threads_started":1}
I20260928 23:14:26.976640 13124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:26.976730 13124 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:26.976905 13124 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:26.977345 12640 ts_manager.cc:194] Registered new tserver with Master: 8597cbd39ad94f8f8c5d0f06c42e15cc (127.0.98.3:38299)
I20260928 23:14:26.977824 12640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54929
I20260928 23:14:26.986321 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:14:26.987016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 12721
I20260928 23:14:26.993383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 12860
I20260928 23:14:26.998656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 12995
I20260928 23:14:27.004735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 12608
I20260928 23:14:27.009155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--webserver_interface=127.0.98.62
--webserver_port=35337
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45897
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:27.087744 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:27.087955 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:27.087976 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:27.089090 13127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:27.089133 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:27.089144 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:27.089157 13127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:27.089181 13127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:27.090807 13127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45897
--ranger_config_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:27.091043 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:27.091239 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:27.095721 13135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.095731 13133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.095746 13132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:27.095894 13127 server_base.cc:1034] running on GCE node
I20260928 23:14:27.096141 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:27.096414 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:27.097551 13127 hybrid_clock.cc:648] HybridClock initialized: now 1790637267097539 us; error 27 us; skew 500 ppm
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:27.099762 13127 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:27.100157 13127 webserver.cc:460] Webserver started at http://127.0.98.62:35337/ using document root <none> and password file <none>
I20260928 23:14:27.100339 13127 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:27.100392 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:27.141021 13127 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:27.141722 13143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.141882 13127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.141968 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "be440a6578254c67b4b9a5d971b90a18"
format_stamp: "Formatted at 2026-09-28 23:14:25 on dist-test-slave-6vgd"
server_key: "731ba58b322019712dfc99a1ca505a62"
server_key_iv: "b73c88f689064e7f184481d62e96249c"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:27.142316 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:27.152341 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:27.153996 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:27.154143 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:27.157812 13127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45897
I20260928 23:14:27.157881 13195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45897 every 8 connection(s)
I20260928 23:14:27.158191 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:27.159384 13196 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:14:27.159992 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Bootstrap starting.
I20260928 23:14:27.163800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13127
I20260928 23:14:27.164038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46117
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45897
--webserver_port=41267
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:27.168073 13196 log.cc:824] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:27.169750 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:27.170001 13196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Bootstrap complete.
I20260928 23:14:27.172145 13196 raft_consensus.cc:348] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:27.172370 13196 raft_consensus.cc:729] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be440a6578254c67b4b9a5d971b90a18, State: Initialized, Role: FOLLOWER
I20260928 23:14:27.172514 13196 consensus_queue.cc:260] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:27.172585 13196 raft_consensus.cc:388] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:27.172631 13196 raft_consensus.cc:482] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:27.172675 13196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:14:27.173671 13196 raft_consensus.cc:504] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:27.173765 13196 leader_election.cc:302] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be440a6578254c67b4b9a5d971b90a18; no voters:
I20260928 23:14:27.173916 13196 leader_election.cc:288] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:14:27.173964 13200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:14:27.174072 13200 raft_consensus.cc:686] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [term 2 LEADER]: Becoming Leader. State: Replica: be440a6578254c67b4b9a5d971b90a18, State: Running, Role: LEADER
I20260928 23:14:27.174170 13196 sys_catalog.cc:564] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:27.174149 13200 consensus_queue.cc:237] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260928 23:14:27.174458 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "be440a6578254c67b4b9a5d971b90a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260928 23:14:27.174465 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be440a6578254c67b4b9a5d971b90a18. Latest consensus state: current_term: 2 leader_uuid: "be440a6578254c67b4b9a5d971b90a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be440a6578254c67b4b9a5d971b90a18" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260928 23:14:27.174547 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:27.174554 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:27.174727 13206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:27.175174 13206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:27.175294 13206 catalog_manager.cc:1259] Loaded cluster ID: 2971b055fa784474899b0e2b67b47f39
I20260928 23:14:27.175318 13206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:27.175606 13196 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 23:14:27.176430 13206 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:27.176596 13206 catalog_manager.cc:5879] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Loaded TSK: 0
I20260928 23:14:27.176800 13206 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:14:27.272727 13198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:27.273001 13198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:27.273041 13198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:27.274752 13198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:27.274812 13198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:27.274854 13198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:27.276983 13198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41267
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:27.277254 13198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:27.277508 13198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:14:27.284196 13198 server_base.cc:1034] running on GCE node
W20260928 23:14:27.284364 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.284451 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.284621 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:27.284886 13198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:27.285156 13198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:27.286314 13198 hybrid_clock.cc:648] HybridClock initialized: now 1790637267286285 us; error 42 us; skew 500 ppm
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:27.289562 13198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:27.290086 13198 webserver.cc:460] Webserver started at http://127.0.98.1:41267/ using document root <none> and password file <none>
I20260928 23:14:27.290333 13198 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:27.290413 13198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:27.318924 13198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 23:14:27.319732 13244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.319887 13198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:27.319969 13198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "927290f7fd8a4a8da2bfb794caad74c5"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "1be44fb9910c3481640b1d6486a32dc7"
server_key_iv: "1d3eb8f7d99d8fd0bf3ae1fe1ea8e237"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:27.320405 13198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:27.340842 13198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:27.343137 13198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:27.343298 13198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:27.343566 13198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:27.343897 13198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:27.343943 13198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.343969 13198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:27.343989 13198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.349848 13198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46117
I20260928 23:14:27.350255 13198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:14:27.351197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13198
I20260928 23:14:27.351457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37347
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45897
--webserver_port=34449
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:27.357882 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46117 every 8 connection(s)
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:27.371498 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.358860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42255 (local address 127.0.98.62:45897)
0928 23:14:27.359483 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:27.359492 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:27.359534 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:14:27.359969 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:27.359978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:27.360037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:27.360154 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:27.361029 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.362460 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.363955 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.364579 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.364659 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.367360 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:27.367365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:27.367365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:27.367377 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:27.369423 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.369920 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.369923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.369924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.370033 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.370327 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.370328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.370328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.370618 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:27.370673 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:27.371132 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 23:14:27.371263 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":71,"threads_started":1}
I20260928 23:14:27.372377 13361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.358505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.1:42255)
0928 23:14:27.359047 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:27.359108 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:27.359713 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:27.360377 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:27.360381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:27.360627 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:27.360903 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.360909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.362588 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.362588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.363859 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.363861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.364784 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.364785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.364918 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.365578 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:27.365588 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:27.367232 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:27.369513 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.369519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.369521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.369824 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.370124 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.370124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.370125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.370254 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.370722 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:27.370728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:27.370904 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 23:14:27.371978 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0928 23:14:27.372157 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":91,"threads_started":1}
I20260928 23:14:27.373529 13358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:27.373621 13358 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.373796 13358 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.374717 13358 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:14:27.374779 13358 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.374835 13358 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:27.375213 13358 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.375283 13358 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.375640 13358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:14:27.465082 13360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:27.465310 13360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:27.465345 13360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:27.466496 13360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:27.466536 13360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:27.466568 13360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:27.467967 13360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34449
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:27.468175 13360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:27.468370 13360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:27.473097 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:27.473245 13360 server_base.cc:1034] running on GCE node
W20260928 23:14:27.473107 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.473409 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:27.473778 13360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:27.474020 13360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:27.475157 13360 hybrid_clock.cc:648] HybridClock initialized: now 1790637267475146 us; error 26 us; skew 500 ppm
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:27.477638 13360 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:27.478070 13360 webserver.cc:460] Webserver started at http://127.0.98.2:34449/ using document root <none> and password file <none>
I20260928 23:14:27.478309 13360 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:27.478370 13360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:27.495671 13360 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:27.497347 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.497502 13360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:27.497591 13360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "0c051bc6fecd4fd0844d983e88242d4a"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "3c36c119435813d284016d675896298e"
server_key_iv: "d774d8212c07011eed23043acb132cee"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:27.498036 13360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:27.513204 13360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:27.515148 13360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:27.515270 13360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:27.515512 13360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:27.515812 13360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:27.515844 13360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.515872 13360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:27.515889 13360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.521909 13360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37347
I20260928 23:14:27.521997 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37347 every 8 connection(s)
I20260928 23:14:27.522370 13360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:27.525321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13360
I20260928 23:14:27.525604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38299
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45897
--webserver_port=33087
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:27.531817 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.523217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49615 (local address 127.0.98.62:45897)
0928 23:14:27.523272 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:27.523274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:27.523516 (+ 242us) server_negotiation.cc:407] Connection header received
0928 23:14:27.524022 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:27.524024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:27.524038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:27.524084 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:27.524635 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.525180 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.526178 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.526381 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.526436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.529015 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:27.529016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:27.529017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:27.529024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:27.530440 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.530793 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.530795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.530795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.530848 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.531122 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.531123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.531123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.531251 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:27.531291 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:27.531645 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 23:14:27.531729 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:14:27.532299 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.523222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.2:49615)
0928 23:14:27.523379 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:27.523418 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:27.523930 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:27.524166 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:27.524170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:27.524351 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:27.524547 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.524552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.525283 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.525284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.526096 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.526098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.526466 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.526467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.526599 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.527274 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:27.527284 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:27.528905 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:27.530514 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.530520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.530523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.530728 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.530901 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.530902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.530902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.531047 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.531331 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:27.531336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:27.531504 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 23:14:27.531907 (+ 403us) client_negotiation.cc:240] Negotiation successful
0928 23:14:27.532077 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260928 23:14:27.532714 13492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:27.532819 13492 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.533026 13492 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.533798 13492 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:14:27.533862 13492 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.533916 13492 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:27.534336 13492 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.534409 13492 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.534772 13492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:14:27.623942 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:27.624223 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:27.624248 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:27.625455 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:27.625497 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:27.625531 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:27.627529 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33087
--tserver_master_addrs=127.0.98.62:45897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:27.627746 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:27.627959 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:27.635095 13501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.635833 13502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:27.635942 13504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:27.636512 13495 server_base.cc:1034] running on GCE node
I20260928 23:14:27.636700 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:27.636930 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:27.638073 13495 hybrid_clock.cc:648] HybridClock initialized: now 1790637267638057 us; error 31 us; skew 500 ppm
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:27.640407 13495 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:27.640796 13495 webserver.cc:460] Webserver started at http://127.0.98.3:33087/ using document root <none> and password file <none>
I20260928 23:14:27.640987 13495 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:27.641057 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:27.655411 13495 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:27.656128 13512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.656314 13495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:27.656392 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc"
format_stamp: "Formatted at 2026-09-28 23:14:26 on dist-test-slave-6vgd"
server_key: "ea6dfda1c1e9435b55c57c0178b27681"
server_key_iv: "f755eb76a2b182b1a29acce9ab60b4f1"
server_key_version: "kuduclusterkey@0"
I20260928 23:14:27.656798 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:27.683588 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:27.685474 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:27.685599 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:27.685825 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:27.686165 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:27.686203 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.686232 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:27.686250 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:27.692206 13495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38299
I20260928 23:14:27.692646 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:14:27.693436 13625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38299 every 8 connection(s)
I20260928 23:14:27.694530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13495
Sep 28 23:14:27 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:27.703902 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.694888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.98.62:45897)
0928 23:14:27.694983 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:27.694985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:27.695144 (+ 159us) server_negotiation.cc:407] Connection header received
0928 23:14:27.695211 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:27.695212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:27.695223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:27.695287 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:27.695953 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.696442 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.697411 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.697625 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.697678 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.698883 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:27.698884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:27.698884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:27.698890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:27.701264 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.703212 (+ 1948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.703214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.703214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.703272 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.703435 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.703436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.703437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.703548 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:27.703698 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:27.703732 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:14:27.703802 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":51,"threads_started":1}
I20260928 23:14:27.708339 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.693958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50067 (local address 127.0.98.62:45897)
0928 23:14:27.694032 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:27.694034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:27.694260 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:14:27.695272 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:27.695274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:27.695284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:27.695312 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:27.696112 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.696805 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.698278 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.698571 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.698641 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.705276 (+ 6635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:27.705277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:27.705278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:27.705285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:27.706780 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.707283 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.707284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.707285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.707336 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:27.707610 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:27.707611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:27.707611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:27.707733 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:27.707775 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:27.708177 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 23:14:27.708251 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 23:14:27.709015 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:27.693950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.98.3:50067)
0928 23:14:27.694111 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:27.694166 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:27.695154 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:27.695496 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:27.695501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:27.695736 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:27.696010 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.696017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.696912 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.696913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.698195 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:27.698198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:27.698674 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:27.698675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:27.698812 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:27.699824 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:27.699833 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:27.705147 (+ 5314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:27.706869 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.706876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.706878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.707197 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.707402 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:27.707402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:27.707403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:27.707530 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:27.707823 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:27.707830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:27.708016 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 23:14:27.708619 (+ 603us) client_negotiation.cc:240] Negotiation successful
0928 23:14:27.708777 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 23:14:27.709646 13626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45897
I20260928 23:14:27.709733 13626 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.709913 13626 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.710533 13626 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 23:14:27.710600 13626 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.710657 13626 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:27.711006 13626 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:27.711073 13626 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
W20260928 23:14:27.711375 13626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 23:14:28 dist-test-slave-6vgd krb5kdc[12164](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790637268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:28.375816 13358 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:28.375974 13358 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:28.376390 13160 ts_manager.cc:194] Registered new tserver with Master: 927290f7fd8a4a8da2bfb794caad74c5 (127.0.98.1:46117)
I20260928 23:14:28.377449 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42255
I20260928 23:14:28.534964 13492 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:28.535100 13492 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:28.535507 13159 ts_manager.cc:194] Registered new tserver with Master: 0c051bc6fecd4fd0844d983e88242d4a (127.0.98.2:37347)
I20260928 23:14:28.536018 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49615
Sep 28 23:14:28 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 23:14:28.711548 13626 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:28.711699 13626 heartbeater.cc:511] Master 127.0.98.62:45897 requested a full tablet report, sending...
I20260928 23:14:28.712049 13160 ts_manager.cc:194] Registered new tserver with Master: 8597cbd39ad94f8f8c5d0f06c42e15cc (127.0.98.3:38299)
I20260928 23:14:28.712553 13160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50067
I20260928 23:14:28.719488 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:14:28.732110 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.720480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.0.98.62:45897)
0928 23:14:28.720822 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.720824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.720989 (+ 165us) server_negotiation.cc:407] Connection header received
0928 23:14:28.721014 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.721014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.721026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.721079 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:28.721819 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.722597 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.723639 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.724007 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.724065 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.725619 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:28.725620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:28.725620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:28.725627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:28.730569 (+ 4942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:28.730993 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:28.730994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:28.730994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:28.731055 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:28.731265 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:28.731266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:28.731266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:28.731399 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:28.731483 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.731915 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.731997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":60,"threads_started":1}
I20260928 23:14:28.734429 13160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:14:28.736618 13160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:14:28.748052 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.739945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37347 (local address 127.0.0.1:50242)
0928 23:14:28.741637 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.741645 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:28.741681 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:28.741920 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:28.741923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:28.741925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:28.741996 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.741998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.743538 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.743539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.745742 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.745743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.747665 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.747667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.747771 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.747795 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 23:14:28.747881 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:14:28.747952 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1679,"thread_start_us":62,"threads_started":1}
I20260928 23:14:28.748548 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.740057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.0.98.2:37347)
0928 23:14:28.741572 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.741577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.741722 (+ 145us) server_negotiation.cc:407] Connection header received
0928 23:14:28.741755 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.741758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.741773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.741832 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:28.742060 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.743442 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.745849 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.746640 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.746705 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.746783 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.748406 (+ 1623us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.748464 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":49,"threads_started":1}
I20260928 23:14:28.750300 13426 tablet_service.cc:1467] Processing CreateTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 (DEFAULT_TABLE table=test-table [id=13d716af0e774e19a0714fdcb431a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:28.750677 13426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aab3ff2925401f8c36d88cdb601a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:28.756300 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.739718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46117 (local address 127.0.0.1:41714)
0928 23:14:28.741334 (+ 1616us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.741377 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:28.741441 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:28.745174 (+ 3733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:28.745175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:28.745177 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:28.745364 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.745366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.753763 (+ 8397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.753764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.754784 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.754786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.756057 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.756058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.756123 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.756128 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:14:28.756155 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 23:14:28.756201 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"mutex_wait_us":135,"thread_start_us":78,"threads_started":1}
I20260928 23:14:28.756722 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.740128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.0.98.1:46117)
0928 23:14:28.744989 (+ 4861us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.744992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.745002 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:14:28.745038 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.745040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.745056 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.745249 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:28.746821 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.749468 (+ 2647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.755195 (+ 5727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.755959 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.756469 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.756573 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.756600 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.756645 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4828,"thread_start_us":62,"threads_started":1}
I20260928 23:14:28.757544 13292 tablet_service.cc:1467] Processing CreateTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 (DEFAULT_TABLE table=test-table [id=13d716af0e774e19a0714fdcb431a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:28.757886 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aab3ff2925401f8c36d88cdb601a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:28.762677 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.740499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.98.3:38299)
0928 23:14:28.744882 (+ 4383us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.744887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.749930 (+ 5043us) server_negotiation.cc:407] Connection header received
0928 23:14:28.749972 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.749975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.749991 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.750056 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:28.750239 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.751597 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.752825 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.753559 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.753627 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.753699 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.762543 (+ 8844us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.762601 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4340,"thread_start_us":55,"threads_started":1}
I20260928 23:14:28.762272 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.740399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38299 (local address 127.0.0.1:40338)
0928 23:14:28.749820 (+ 9421us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.749831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:28.749871 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:28.750132 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:28.750133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:28.750134 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:28.750167 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.750168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.751688 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.751689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.752734 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.752737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.761998 (+ 9261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.761999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.762082 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.762086 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:14:28.762135 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:14:28.762185 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9408,"thread_start_us":71,"threads_started":1}
I20260928 23:14:28.763936 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 (DEFAULT_TABLE table=test-table [id=13d716af0e774e19a0714fdcb431a8bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:28.764264 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66aab3ff2925401f8c36d88cdb601a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:28.769392 13659 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Bootstrap starting.
I20260928 23:14:28.770231 13659 tablet_bootstrap.cc:654] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:28.770385 13660 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Bootstrap starting.
I20260928 23:14:28.770716 13659 log.cc:824] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:28.771049 13660 tablet_bootstrap.cc:654] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:28.771364 13660 log.cc:824] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:28.771564 13659 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: No bootstrap required, opened a new log
I20260928 23:14:28.771620 13659 ts_tablet_manager.cc:1397] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:14:28.774204 13659 raft_consensus.cc:348] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.774371 13659 raft_consensus.cc:374] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:28.774399 13659 raft_consensus.cc:729] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927290f7fd8a4a8da2bfb794caad74c5, State: Initialized, Role: FOLLOWER
I20260928 23:14:28.774732 13659 consensus_queue.cc:260] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.775101 13659 ts_tablet_manager.cc:1428] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:14:28.775261 13358 heartbeater.cc:503] Master 127.0.98.62:45897 was elected leader, sending a full tablet report...
I20260928 23:14:28.775341 13661 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Bootstrap starting.
I20260928 23:14:28.776000 13661 tablet_bootstrap.cc:654] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:28.776302 13661 log.cc:824] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:28.777407 13660 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: No bootstrap required, opened a new log
I20260928 23:14:28.777525 13660 ts_tablet_manager.cc:1397] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 23:14:28.780165 13660 raft_consensus.cc:348] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.780403 13660 raft_consensus.cc:374] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:28.780491 13660 raft_consensus.cc:729] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c051bc6fecd4fd0844d983e88242d4a, State: Initialized, Role: FOLLOWER
I20260928 23:14:28.780695 13660 consensus_queue.cc:260] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.781047 13660 ts_tablet_manager.cc:1428] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:14:28.781199 13492 heartbeater.cc:503] Master 127.0.98.62:45897 was elected leader, sending a full tablet report...
I20260928 23:14:28.788339 13661 tablet_bootstrap.cc:492] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: No bootstrap required, opened a new log
I20260928 23:14:28.788443 13661 ts_tablet_manager.cc:1397] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260928 23:14:28.791162 13661 raft_consensus.cc:348] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.791380 13661 raft_consensus.cc:374] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:28.791424 13661 raft_consensus.cc:729] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8597cbd39ad94f8f8c5d0f06c42e15cc, State: Initialized, Role: FOLLOWER
I20260928 23:14:28.791574 13661 consensus_queue.cc:260] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.791869 13661 ts_tablet_manager.cc:1428] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 23:14:28.791996 13626 heartbeater.cc:503] Master 127.0.98.62:45897 was elected leader, sending a full tablet report...
I20260928 23:14:28.854260 13664 raft_consensus.cc:482] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:28.854400 13664 raft_consensus.cc:504] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.854827 13664 leader_election.cc:288] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8597cbd39ad94f8f8c5d0f06c42e15cc (127.0.98.3:38299), 0c051bc6fecd4fd0844d983e88242d4a (127.0.98.2:37347)
Sep 28 23:14:28 dist-test-slave-6vgd krb5kdc[12164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260928 23:14:28.859242 13359 tablet.cc:2369] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:28.863780 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.855579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59161 (local address 127.0.98.3:38299)
0928 23:14:28.855623 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.855624 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.855632 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:28.855658 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.855659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.855668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.855713 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:28.856472 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.857252 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.862801 (+ 5549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.863385 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.863433 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.863446 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.863659 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.863704 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:14:28.864045 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.855095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38299 (local address 127.0.98.1:59161)
0928 23:14:28.855265 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.855276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:28.855465 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:28.855767 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:28.855768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:28.855770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:28.856374 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.856379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.861408 (+ 5029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.861409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.862693 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.862697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.863505 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.863506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.863581 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.863584 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:28.863903 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 23:14:28.863953 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260928 23:14:28.865015 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aab3ff2925401f8c36d88cdb601a70" candidate_uuid: "927290f7fd8a4a8da2bfb794caad74c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" is_pre_election: true
I20260928 23:14:28.865172 13580 raft_consensus.cc:2454] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927290f7fd8a4a8da2bfb794caad74c5 in term 0.
I20260928 23:14:28.865504 13245 leader_election.cc:302] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8597cbd39ad94f8f8c5d0f06c42e15cc, 927290f7fd8a4a8da2bfb794caad74c5; no voters:
I20260928 23:14:28.865638 13664 raft_consensus.cc:2781] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:14:28.865680 13664 raft_consensus.cc:482] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:14:28.865702 13664 raft_consensus.cc:3037] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:28.866581 13664 raft_consensus.cc:504] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.866725 13664 leader_election.cc:288] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [CANDIDATE]: Term 1 election: Requested vote from peers 8597cbd39ad94f8f8c5d0f06c42e15cc (127.0.98.3:38299), 0c051bc6fecd4fd0844d983e88242d4a (127.0.98.2:37347)
I20260928 23:14:28.866914 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aab3ff2925401f8c36d88cdb601a70" candidate_uuid: "927290f7fd8a4a8da2bfb794caad74c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc"
I20260928 23:14:28.866981 13580 raft_consensus.cc:3037] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:28.867812 13580 raft_consensus.cc:2454] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927290f7fd8a4a8da2bfb794caad74c5 in term 1.
I20260928 23:14:28.868142 13245 leader_election.cc:302] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8597cbd39ad94f8f8c5d0f06c42e15cc, 927290f7fd8a4a8da2bfb794caad74c5; no voters:
I20260928 23:14:28.868234 13664 raft_consensus.cc:2781] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:28.868373 13664 raft_consensus.cc:686] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [term 1 LEADER]: Becoming Leader. State: Replica: 927290f7fd8a4a8da2bfb794caad74c5, State: Running, Role: LEADER
I20260928 23:14:28.868465 13664 consensus_queue.cc:237] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } }
I20260928 23:14:28.869329 13160 catalog_manager.cc:5495] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 927290f7fd8a4a8da2bfb794caad74c5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "927290f7fd8a4a8da2bfb794caad74c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "927290f7fd8a4a8da2bfb794caad74c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 } health_report { overall_health: UNKNOWN } } }
I20260928 23:14:28.874284 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.855399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57669 (local address 127.0.98.2:37347)
0928 23:14:28.855439 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.855441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.864346 (+ 8905us) server_negotiation.cc:407] Connection header received
0928 23:14:28.864446 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.864447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.864461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.864490 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:28.864676 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.865319 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.872679 (+ 7360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.873317 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.873383 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.873395 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.874149 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.874213 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:14:28.874522 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.855332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37347 (local address 127.0.98.1:57669)
0928 23:14:28.864286 (+ 8954us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.864296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:28.864376 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:28.864549 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:28.864550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:28.864551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:28.864605 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.864606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.871370 (+ 6764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.871371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.872577 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:28.872580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.873982 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.873985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:28.874059 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.874064 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:14:28.874405 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 23:14:28.874449 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":65,"threads_started":1}
I20260928 23:14:28.874882 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aab3ff2925401f8c36d88cdb601a70" candidate_uuid: "927290f7fd8a4a8da2bfb794caad74c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c051bc6fecd4fd0844d983e88242d4a" is_pre_election: true
I20260928 23:14:28.875042 13446 raft_consensus.cc:2454] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 927290f7fd8a4a8da2bfb794caad74c5 in term 0.
I20260928 23:14:28.875162 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66aab3ff2925401f8c36d88cdb601a70" candidate_uuid: "927290f7fd8a4a8da2bfb794caad74c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c051bc6fecd4fd0844d983e88242d4a"
I20260928 23:14:28.875233 13445 raft_consensus.cc:3037] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:28.876132 13445 raft_consensus.cc:2454] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 927290f7fd8a4a8da2bfb794caad74c5 in term 1.
I20260928 23:14:28.888599 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.885195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.0.98.1:46117)
0928 23:14:28.885276 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:28.885278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:28.885286 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:28.885322 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:28.885323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:28.885345 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:28.885409 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:14:28.885580 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.886325 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.887488 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:28.887805 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:28.887866 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:28.888064 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:14:28.888199 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:14:28.888269 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:28.888498 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 23:14:28.888519 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 23:14:28.890552 13445 raft_consensus.cc:1261] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 1 FOLLOWER]: Refusing update from remote peer 927290f7fd8a4a8da2bfb794caad74c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:28.890597 13580 raft_consensus.cc:1261] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 1 FOLLOWER]: Refusing update from remote peer 927290f7fd8a4a8da2bfb794caad74c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:28.890887 13671 consensus_queue.cc:1035] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c051bc6fecd4fd0844d983e88242d4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:28.891010 13664 consensus_queue.cc:1035] T 66aab3ff2925401f8c36d88cdb601a70 P 927290f7fd8a4a8da2bfb794caad74c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8597cbd39ad94f8f8c5d0f06c42e15cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:28.894589 13675 mvcc.cc:205] Tried to move back new op lower bound from 7334450253372571648 to 7334450253285830656. Current Snapshot: MvccSnapshot[applied={T|T < 7334450253372571648}]
I20260928 23:14:28.895514 13674 mvcc.cc:205] Tried to move back new op lower bound from 7334450253372571648 to 7334450253285830656. Current Snapshot: MvccSnapshot[applied={T|T < 7334450253372571648}]
I20260928 23:14:28.895677 13676 mvcc.cc:205] Tried to move back new op lower bound from 7334450253372571648 to 7334450253285830656. Current Snapshot: MvccSnapshot[applied={T|T < 7334450253372571648}]
I20260928 23:14:28.898252 13160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:28.898350 13160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:28.899129 13160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be440a6578254c67b4b9a5d971b90a18: Sending DeleteTablet for 3 replicas of tablet 66aab3ff2925401f8c36d88cdb601a70
I20260928 23:14:28.899497 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40338
I20260928 23:14:28.899523 13426 tablet_service.cc:1514] Processing DeleteTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50242
I20260928 23:14:28.899922 13292 tablet_service.cc:1514] Processing DeleteTablet for tablet 66aab3ff2925401f8c36d88cdb601a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41714
I20260928 23:14:28.900146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13198
I20260928 23:14:28.900158 13684 tablet_replica.cc:330] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: stopping tablet replica
I20260928 23:14:28.900738 13684 raft_consensus.cc:2229] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:28.900916 13684 raft_consensus.cc:2258] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:28.901414 13684 ts_tablet_manager.cc:1905] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:28.901949 13683 tablet_replica.cc:330] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: stopping tablet replica
I20260928 23:14:28.902101 13683 raft_consensus.cc:2229] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:28.902715 13683 raft_consensus.cc:2258] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:28.903281 13683 ts_tablet_manager.cc:1905] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:28.905805 13683 ts_tablet_manager.cc:1918] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:28.905931 13683 log.cc:1196] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/66aab3ff2925401f8c36d88cdb601a70
I20260928 23:14:28.906345 13684 ts_tablet_manager.cc:1918] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:28.906425 13684 log.cc:1196] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/66aab3ff2925401f8c36d88cdb601a70
I20260928 23:14:28.906520 13683 ts_tablet_manager.cc:1939] T 66aab3ff2925401f8c36d88cdb601a70 P 8597cbd39ad94f8f8c5d0f06c42e15cc: Deleting consensus metadata
I20260928 23:14:28.906674 13684 ts_tablet_manager.cc:1939] T 66aab3ff2925401f8c36d88cdb601a70 P 0c051bc6fecd4fd0844d983e88242d4a: Deleting consensus metadata
I20260928 23:14:28.907081 13144 catalog_manager.cc:4841] TS 8597cbd39ad94f8f8c5d0f06c42e15cc (127.0.98.3:38299): tablet 66aab3ff2925401f8c36d88cdb601a70 (table test-table [id=13d716af0e774e19a0714fdcb431a8bd]) successfully deleted
I20260928 23:14:28.907076 13145 catalog_manager.cc:4841] TS 0c051bc6fecd4fd0844d983e88242d4a (127.0.98.2:37347): tablet 66aab3ff2925401f8c36d88cdb601a70 (table test-table [id=13d716af0e774e19a0714fdcb431a8bd]) successfully deleted
W20260928 23:14:28.907203 13147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46117 (error 108)
I20260928 23:14:28.907635 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:28.907523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46117 (local address 127.0.0.1:41732)
0928 23:14:28.907573 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:28.907605 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 23:14:28.908429 13147 catalog_manager.cc:4584] TS 927290f7fd8a4a8da2bfb794caad74c5 (127.0.98.1:46117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66aab3ff2925401f8c36d88cdb601a70: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46117: connect: Connection refused (error 111)
I20260928 23:14:28.908767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13360
I20260928 23:14:28.913940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13495
I20260928 23:14:28.919343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13127
I20260928 23:14:28.929838 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 23:14:29.276660 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 23:14:29.276753 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 23:14:29.620728 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 23:14:29.620 UTC [12194] LOG: received smart shutdown request
2026-09-28 23:14:29.624 UTC [12194] LOG: background worker "logical replication launcher" (PID 12203) exited with exit code 1
2026-09-28 23:14:29.624 UTC [12198] LOG: shutting down
2026-09-28 23:14:29.772 UTC [12194] LOG: database system is shut down
2026-09-28T23:14:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44324 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:14:29 dist-test-slave-6vgd krb5kdc[13696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:14:29 dist-test-slave-6vgd krb5kdc[13696](info): set up 2 sockets
Sep 28 23:14:29 dist-test-slave-6vgd krb5kdc[13696](info): commencing operation
krb5kdc: starting...
W20260928 23:14:30.820801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:14:30 dist-test-slave-6vgd krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:14:30.933002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:31.011014 13709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:31.011225 13709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:31.011255 13709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:31.012408 13709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:31.012449 13709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:31.012466 13709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:31.012483 13709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:31.012497 13709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:31.013990 13709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:31.014231 13709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:31.014422 13709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:31.018999 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:31.019012 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:31.019090 13709 server_base.cc:1034] running on GCE node
W20260928 23:14:31.018998 13717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:31.019436 13709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:31.019718 13709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:31.020869 13709 hybrid_clock.cc:648] HybridClock initialized: now 1790637271020857 us; error 26 us; skew 500 ppm
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:31.028448 13709 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:31.028887 13709 webserver.cc:460] Webserver started at http://127.0.98.62:42225/ using document root <none> and password file <none>
I20260928 23:14:31.029081 13709 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:31.029129 13709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:31.029274 13709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:31.030300 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "8ba8aaa563f642ec9f531116ed2205fa"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "75c435cc5cf16798b5c1ec5675eb1e28"
server_key_iv: "48fd5a9341a9866f85f72759257697e1"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.030629 13709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ba8aaa563f642ec9f531116ed2205fa"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "75c435cc5cf16798b5c1ec5675eb1e28"
server_key_iv: "48fd5a9341a9866f85f72759257697e1"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.031813 13709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:14:31.032423 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:31.032580 13709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:31.032660 13709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "8ba8aaa563f642ec9f531116ed2205fa"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "75c435cc5cf16798b5c1ec5675eb1e28"
server_key_iv: "48fd5a9341a9866f85f72759257697e1"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.032720 13709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:31.045156 13709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:31.046801 13709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:31.046932 13709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:31.050606 13709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33581
I20260928 23:14:31.050647 13776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33581 every 8 connection(s)
I20260928 23:14:31.050953 13709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:31.051759 13777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:31.053313 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.057231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13709
I20260928 23:14:31.057356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:14:31.057555 392 external_mini_cluster.cc:1428] Setting key 5fee1fe676db4db29febc67c5fc13402
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:31.062395 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.053934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.0.1:42232)
0928 23:14:31.054111 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.054149 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.054591 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.054805 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.054807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.055055 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.055216 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.055220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.056132 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.056133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.056783 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.056784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.057267 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.057268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.057348 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.057823 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.057832 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.059283 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.060982 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.060988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.060990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.061179 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.061497 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.061497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.061498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.061582 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.061991 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.061996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.062062 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.062128 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.062243 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260928 23:14:31.062420 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.053950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.98.62:33581)
0928 23:14:31.054154 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.054158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.054227 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:14:31.054701 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.054703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.054719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.054791 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.055280 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.056068 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.056839 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.057134 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.057205 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.059359 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.059364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.059365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.059371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.060894 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.061308 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.061311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.061311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.061428 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.061644 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.061645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.061645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.061921 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.062004 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.062177 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.062320 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":183,"spinlock_wait_cycles":8960,"thread_start_us":88,"threads_started":1}
I20260928 23:14:31.063186 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.063475 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.063379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47466)
0928 23:14:31.063416 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.063456 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:14:31.063609 13726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
I20260928 23:14:31.063982 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.063905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47478)
0928 23:14:31.063945 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.063967 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:14:31.064070 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38491: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:14:31.079964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38491
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:31.113286 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } attempt: 1
I20260928 23:14:31.113719 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.113603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47488)
0928 23:14:31.113656 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.113689 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:14:31.114106 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.114017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47502)
0928 23:14:31.114066 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.114089 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:14:31.114204 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38491: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
W20260928 23:14:31.158509 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:31.158716 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:31.158746 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:31.159861 13784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:31.159904 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:31.159920 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:31.159936 13784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:31.159950 13784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:31.161486 13784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:31.161743 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:31.161979 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:31.166498 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:31.166491 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:31.166498 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:31.166679 13784 server_base.cc:1034] running on GCE node
I20260928 23:14:31.166939 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:31.167236 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:31.168393 13784 hybrid_clock.cc:648] HybridClock initialized: now 1790637271168380 us; error 27 us; skew 500 ppm
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:31.170881 13784 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:14:31.171303 13784 webserver.cc:460] Webserver started at http://127.0.98.61:36945/ using document root <none> and password file <none>
I20260928 23:14:31.171519 13784 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:31.171602 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:31.171720 13784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:31.172755 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data/instance:
uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "8275f8b3130dc7dc1109794878606cee"
server_key_iv: "87ee2e64ac765fd25af724f80e1f2ef9"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.173079 13784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal/instance:
uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "8275f8b3130dc7dc1109794878606cee"
server_key_iv: "87ee2e64ac765fd25af724f80e1f2ef9"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.174312 13784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:31.174955 13799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:31.175109 13784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:31.175186 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "8275f8b3130dc7dc1109794878606cee"
server_key_iv: "87ee2e64ac765fd25af724f80e1f2ef9"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.175248 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:31.182341 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } attempt: 2
I20260928 23:14:31.182739 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.182629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47508)
0928 23:14:31.182679 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.182711 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:14:31.183080 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.182986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47512)
0928 23:14:31.183026 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.183050 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:14:31.183157 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38491: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38491: connect: Connection refused (error 111)
I20260928 23:14:31.198958 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:31.200707 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:31.200840 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:31.204523 13784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38491
I20260928 23:14:31.204568 13851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38491 every 8 connection(s)
I20260928 23:14:31.204882 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:14:31.205688 13852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:31.207178 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:31.214370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13784
I20260928 23:14:31.214485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-1/wal/instance
I20260928 23:14:31.214668 392 external_mini_cluster.cc:1428] Setting key a85fd2993927edf63b235362524a46c4
I20260928 23:14:31.215122 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.207824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.0.98.62:33581)
0928 23:14:31.207867 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.207868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.208042 (+ 174us) server_negotiation.cc:407] Connection header received
0928 23:14:31.208473 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.208474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.208483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.208534 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.209064 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.209596 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.210475 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.210675 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.210719 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.212555 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.212556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.212557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.212564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.213815 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.214152 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.214152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.214153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.214202 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.214484 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.214485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.214485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.214623 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.214675 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.214983 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.215048 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:14:31.215332 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.207790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.0.1:42234)
0928 23:14:31.207929 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.207968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.208404 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.208584 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.208587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.208820 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.209002 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.209005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.209653 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.209654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.210416 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.210418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.210725 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.210725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.210823 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.211228 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.211236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.212466 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.213877 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.213883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.213885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.214096 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.214255 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.214256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.214256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.214389 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.214681 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.214689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.214851 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.214960 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.215119 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 23:14:31.215914 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 23:14:31.222645 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.216154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47522)
0928 23:14:31.216190 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.216202 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.216245 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.216438 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.216439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.216580 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.216633 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.216635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.217469 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.217470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.218204 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.218207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.218636 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.218637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.218709 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.219028 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.219032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.220245 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.221793 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.221794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.221794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.221970 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.222123 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.222124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.222124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.222167 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.222395 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.222396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.222430 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.222489 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.222583 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 23:14:31.222756 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.216186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.98.61:38491)
0928 23:14:31.216316 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.216321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.216338 (+ 17us) server_negotiation.cc:407] Connection header received
0928 23:14:31.216359 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.216360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.216370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.216425 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.216693 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.217405 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.218273 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.218547 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.218612 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.220321 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.220329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.220329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.220334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.221726 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.222029 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.222031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.222031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.222080 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.222229 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.222229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.222230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.222351 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.222435 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.222519 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.222643 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260928 23:14:31.223198 13852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.223510 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.223393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41804)
0928 23:14:31.223452 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.223492 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 23:14:31.223634 13802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
I20260928 23:14:31.223965 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.223880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41820)
0928 23:14:31.223922 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.223944 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
W20260928 23:14:31.224041 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:14:31.237030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44287
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:31.273277 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } attempt: 1
I20260928 23:14:31.273736 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.273620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41834)
0928 23:14:31.273675 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.273706 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:14:31.274070 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.273983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41836)
0928 23:14:31.274021 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.274044 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:14:31.274168 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
I20260928 23:14:31.274273 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } attempt: 3
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:14:31.280752 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.274578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.0.98.61:38491)
0928 23:14:31.274615 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.274616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.274631 (+ 15us) server_negotiation.cc:407] Connection header received
0928 23:14:31.274709 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.274710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.274720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.274764 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.275046 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.275548 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.276218 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.276432 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.276485 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.278466 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.278467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.278467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.278473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.279807 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.280152 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.280153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.280153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.280200 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.280355 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.280356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.280356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.280451 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.280494 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.280638 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.280694 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:14:31.280808 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.274541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47528)
0928 23:14:31.274582 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.274593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.274648 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.274767 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.274768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.274920 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.274982 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.274984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.275636 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.275637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.276151 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.276152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.276489 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.276490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.276566 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.276953 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.276958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.278381 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.279885 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.279886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.279887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.280089 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.280250 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.280251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.280251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.280310 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.280497 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.280498 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.280547 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.280621 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.280714 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:14:31.281185 13777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.281483 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.281412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41846)
0928 23:14:31.281441 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.281466 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 23:14:31.281718 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.281660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41856)
0928 23:14:31.281684 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.281704 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 23:14:31.281792 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
I20260928 23:14:31.312960 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } attempt: 1
I20260928 23:14:31.313407 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.313297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41860)
0928 23:14:31.313349 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.313378 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:14:31.313750 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.313672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41872)
0928 23:14:31.313709 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.313733 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:14:31.313830 13777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
W20260928 23:14:31.316460 13859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:31.316643 13859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:31.316673 13859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:31.317824 13859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:31.317863 13859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:31.317881 13859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:31.317898 13859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:31.317921 13859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:31.319375 13859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581,127.0.98.61:38491,127.0.98.60:44287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:31.319594 13859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:31.319787 13859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:31.324062 13867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:31.324178 13859 server_base.cc:1034] running on GCE node
W20260928 23:14:31.324309 13865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:31.324328 13864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:31.324573 13859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:31.324812 13859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:31.325933 13859 hybrid_clock.cc:648] HybridClock initialized: now 1790637271325923 us; error 24 us; skew 500 ppm
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:31.328123 13859 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:14:31.328497 13859 webserver.cc:460] Webserver started at http://127.0.98.60:42149/ using document root <none> and password file <none>
I20260928 23:14:31.328698 13859 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:31.328745 13859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:31.328866 13859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:31.329968 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data/instance:
uuid: "631fdb2b723341a5b4f15673f8567454"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "044c5a6720c6dfb817a538fd507a69d7"
server_key_iv: "4cf30db53d35924492be8a6c676ffe0a"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.330295 13859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal/instance:
uuid: "631fdb2b723341a5b4f15673f8567454"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "044c5a6720c6dfb817a538fd507a69d7"
server_key_iv: "4cf30db53d35924492be8a6c676ffe0a"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.331477 13859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:14:31.332152 13874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:31.332309 13859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:31.332388 13859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "631fdb2b723341a5b4f15673f8567454"
format_stamp: "Formatted at 2026-09-28 23:14:31 on dist-test-slave-6vgd"
server_key: "044c5a6720c6dfb817a538fd507a69d7"
server_key_iv: "4cf30db53d35924492be8a6c676ffe0a"
server_key_version: "encryptionkey@0"
I20260928 23:14:31.332445 13859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:31.342303 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } attempt: 2
I20260928 23:14:31.342710 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.342596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41886)
0928 23:14:31.342648 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.342678 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:14:31.343034 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.342969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41894)
0928 23:14:31.342999 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.343019 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 23:14:31.343103 13852 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44287: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44287: connect: Connection refused (error 111)
I20260928 23:14:31.359689 13859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:31.361478 13859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:31.361591 13859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:31.365540 13859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44287
I20260928 23:14:31.365587 13926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44287 every 8 connection(s)
I20260928 23:14:31.365913 13859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:14:31.366734 13927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:31.368285 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.371043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 13859
I20260928 23:14:31.371145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790637099107577-392-0/minicluster-data/master-2/wal/instance
I20260928 23:14:31.371311 392 external_mini_cluster.cc:1428] Setting key 2e66704d0aecf5923d8f12d77a5043fd
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:14:31.377378 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.368990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.0.98.62:33581)
0928 23:14:31.369044 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.369046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.369272 (+ 226us) server_negotiation.cc:407] Connection header received
0928 23:14:31.369805 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.369807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.369818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.369873 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.370509 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.371000 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.372276 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.372463 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.372522 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.374652 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.374654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.374654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.374662 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.375975 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.376353 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.376354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.376355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.376407 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.376644 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.376646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.376646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.376754 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.376808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.377187 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.377290 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:14:31.377537 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.368954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.0.1:42238)
0928 23:14:31.369135 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.369173 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.369708 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.369962 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.369966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.370215 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.370411 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.370415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.371076 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.371077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.372210 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.372212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.372544 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.372545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.372670 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.373151 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.373160 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.374565 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.376063 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.376070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.376072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.376287 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.376489 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.376490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.376490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.376588 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.376833 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.376838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.377013 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.377148 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.377323 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260928 23:14:31.378064 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.379845 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.371677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.0.98.60:44287)
0928 23:14:31.371822 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.371827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.371866 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:14:31.371921 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.371924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.371950 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.372018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.372520 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.373406 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.374215 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.374590 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.374656 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.376814 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.376821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.376821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.376827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.378841 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.379149 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.379151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.379152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.379209 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.379349 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.379350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.379350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.379500 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.379630 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.379725 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.379780 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:14:31.384240 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.378311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38491 (local address 127.0.0.1:47538)
0928 23:14:31.378357 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.378367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.378413 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.378533 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.378534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.378702 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.378733 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.378734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.379480 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.379481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.380122 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.380124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.380505 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.380506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.380570 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.380919 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.380924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.382005 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.383399 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.383400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.383400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.383590 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.383751 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.383752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.383752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.383809 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.384032 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.384033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.384067 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.384116 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.384181 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:14:31.384313 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.378365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.61:38491)
0928 23:14:31.378415 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.378417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.378447 (+ 30us) server_negotiation.cc:407] Connection header received
0928 23:14:31.378470 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.378471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.378481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.378522 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.378798 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.379412 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.380196 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.380402 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.380459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.382090 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.382091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.382091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.382097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.383325 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.383656 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.383657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.383657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.383702 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.383861 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.383862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.383862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.383969 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.384030 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.384152 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.384222 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:31.384603 13927 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:14:31.387360 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.380725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.98.62:33581)
0928 23:14:31.380759 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.380761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.380768 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:14:31.380790 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.380790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.380797 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.380846 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.381288 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.381754 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.382474 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.382665 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.382718 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.384916 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.384917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.384917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.384924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.386349 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.386648 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.386649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.386649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.386706 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.386866 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.386867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.386867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.386973 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.387167 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.387200 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.387271 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:14:31.388948 13777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } attempt: 2
I20260928 23:14:31.391281 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.384832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41922)
0928 23:14:31.384870 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.384879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.384913 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.385009 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.385010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.385140 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.385173 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.385175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.385879 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.385880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.386574 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.386576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.386936 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.386937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.386988 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.387346 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.387350 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.388635 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.390276 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.390277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.390278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.390471 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.390671 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.390671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.390672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.390725 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.390974 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.390975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.391015 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.391092 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.391174 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:14:31.391356 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.384881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.98.60:44287)
0928 23:14:31.384914 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.384915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.384923 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:31.384960 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.384960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.384968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.385003 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.385252 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.385809 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.386639 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.386874 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.386924 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.388700 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.388701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.388702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.388708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.390207 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.390538 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.390539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.390540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.390612 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.390787 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.390788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.390788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.390914 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.390968 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.391134 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.391227 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":17}
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:14:31.393162 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: Bootstrap starting.
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:14:31.393905 13927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:31.394157 13927 log.cc:824] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:31.394867 13927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: No bootstrap required, opened a new log
I20260928 23:14:31.395534 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.387981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.0.98.61:38491)
0928 23:14:31.388016 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.388018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.388025 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:14:31.388082 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.388082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.388089 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.388130 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.388634 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.389304 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.390225 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.390443 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.390473 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.392855 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.392856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.392857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.392864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.394409 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.394775 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.394776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.394776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.394835 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.395035 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.395036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.395037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.395171 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.395340 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.395376 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.395448 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:14:31.396454 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.389173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41928)
0928 23:14:31.389297 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.389307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.389349 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.389512 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.389513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.389686 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.389722 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.389723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.390841 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.390842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.391568 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.391571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.392052 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.392053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.392118 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.392475 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.392479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.393751 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.395433 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.395434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.395434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.395631 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.395811 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.395812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.395813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.395868 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.396161 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.396162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.396209 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.396269 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.396361 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 23:14:31.396538 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.389269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.98.60:44287)
0928 23:14:31.389412 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.389413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.389419 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:14:31.389435 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.389436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.389445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.389506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.389804 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.390752 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.391739 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.391989 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.392051 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.393824 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.393826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.393826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.393833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.395351 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.395695 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.395696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.395696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.395751 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.395919 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.395920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.395920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.396082 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.396164 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.396348 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.396418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 23:14:31.396854 13927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.397006 13927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:31.397042 13927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631fdb2b723341a5b4f15673f8567454, State: Initialized, Role: FOLLOWER
I20260928 23:14:31.397168 13927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.397539 13927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:31.397619 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:14:31.397786 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: Bootstrap starting.
I20260928 23:14:31.398418 13777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:31.398669 13777 log.cc:824] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: Log is configured to *not* fsync() on all Append() calls
W20260928 23:14:31.399555 13949 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:14:31.399618 13949 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:14:31.400272 13777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: No bootstrap required, opened a new log
I20260928 23:14:31.399680 13859 master_runner.cc:186] Error getting master registration for 127.0.98.62:33581: OK, instance_id { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" instance_seqno: 1790637271049550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:14:31.401827 13859 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:14:31.402225 13777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.402348 13777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:31.402382 13777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba8aaa563f642ec9f531116ed2205fa, State: Initialized, Role: FOLLOWER
I20260928 23:14:31.402498 13777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.402784 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.402797 13777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:31.402925 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.406164 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.399109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.98.62:33581)
0928 23:14:31.399142 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.399144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.399342 (+ 198us) server_negotiation.cc:407] Connection header received
0928 23:14:31.399400 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.399401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.399409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.399458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.400193 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.400878 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.401865 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.402073 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.402105 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.403477 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.403478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.403479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.403485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.404962 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.405369 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.405370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.405370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.405428 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.405623 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.405624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.405625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.405740 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.405818 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.406007 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.406080 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:14:31.405339 13709 master_runner.cc:186] Error getting master registration for 127.0.98.61:38491: OK, instance_id { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" instance_seqno: 1790637271203501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:14:31.408699 13709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:14:31.409305 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.409408 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.410066 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.399229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.98.60:44287)
0928 23:14:31.399263 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.399265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.403125 (+ 3860us) server_negotiation.cc:407] Connection header received
0928 23:14:31.403174 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.403175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.403185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.403240 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.404080 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.405043 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.405964 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.406263 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.406308 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.407821 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.407823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.407823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.407830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.409033 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.409384 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.409386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.409386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.409445 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.409608 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.409609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.409609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.409730 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.409812 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.409910 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.409981 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:14:31.410914 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:14:31.410967 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:14:31.412844 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.399569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.0.98.61:38491)
0928 23:14:31.399616 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.399618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.403258 (+ 3640us) server_negotiation.cc:407] Connection header received
0928 23:14:31.403349 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.403350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.403360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.403390 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.404032 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.404703 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.405832 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.406111 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.406151 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.410117 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.410118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.410118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.410125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.411665 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.412059 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.412061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.412061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.412121 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.412312 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.412313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.412314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.412439 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.412528 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.412680 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.412760 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:14:31.434249 13852 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } attempt: 3
Sep 28 23:14:31 dist-test-slave-6vgd krb5kdc[13696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:14:31.442267 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.434557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44287 (local address 127.0.0.1:41950)
0928 23:14:31.434609 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:31.434621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:31.434658 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:31.434827 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:31.434828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:31.435024 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:31.435076 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.435078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.435999 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.436001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.436847 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:31.436850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.437348 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.437349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:31.437417 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.437915 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:31.437920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:31.439496 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:31.441245 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.441246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.441246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.441462 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.441649 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:31.441650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:31.441650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:31.441714 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:31.441953 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:31.441954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:31.441999 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 23:14:31.442069 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:14:31.442162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:14:31.442298 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:31.434609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.0.98.60:44287)
0928 23:14:31.434667 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:31.434669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:31.434681 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:14:31.434708 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:31.434709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:31.434744 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:31.434803 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:31.435152 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.435880 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.436947 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:31.437282 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:31.437347 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:31.439586 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:31.439587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:31.439587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:31.439595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:31.441145 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.441532 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.441533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.441533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.441590 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:31.441767 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:31.441768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:31.441768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:31.441897 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:31.441959 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:31.442113 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 23:14:31.442203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:14:31.444240 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: Bootstrap starting.
I20260928 23:14:31.444953 13852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:31.445312 13852 log.cc:824] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:31.454015 13852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: No bootstrap required, opened a new log
I20260928 23:14:31.456355 13852 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.456506 13852 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:31.456538 13852 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1de2dee8c1c410ba3fb9a2b8739e28b, State: Initialized, Role: FOLLOWER
I20260928 23:14:31.456694 13852 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.456971 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.457043 13852 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:31.457130 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:14:31.458971 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:14:31.459059 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:14:31.461256 13784 master_runner.cc:263] No leader master found from master f1de2dee8c1c410ba3fb9a2b8739e28b
I20260928 23:14:31.461299 13784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:14:31.545151 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:31.545286 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.545537 13969 leader_election.cc:288] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ba8aaa563f642ec9f531116ed2205fa (127.0.98.62:33581), 631fdb2b723341a5b4f15673f8567454 (127.0.98.60:44287)
I20260928 23:14:31.545759 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8aaa563f642ec9f531116ed2205fa" is_pre_election: true
I20260928 23:14:31.545915 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1de2dee8c1c410ba3fb9a2b8739e28b in term 0.
I20260928 23:14:31.545902 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631fdb2b723341a5b4f15673f8567454" is_pre_election: true
I20260928 23:14:31.546053 13901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1de2dee8c1c410ba3fb9a2b8739e28b in term 0.
I20260928 23:14:31.546248 13802 leader_election.cc:302] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631fdb2b723341a5b4f15673f8567454, f1de2dee8c1c410ba3fb9a2b8739e28b; no voters:
I20260928 23:14:31.546381 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:14:31.546424 13969 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:14:31.546445 13969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:31.547082 13969 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.547201 13969 leader_election.cc:288] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [CANDIDATE]: Term 1 election: Requested vote from peers 8ba8aaa563f642ec9f531116ed2205fa (127.0.98.62:33581), 631fdb2b723341a5b4f15673f8567454 (127.0.98.60:44287)
I20260928 23:14:31.547360 13901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631fdb2b723341a5b4f15673f8567454"
I20260928 23:14:31.547371 13751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ba8aaa563f642ec9f531116ed2205fa"
I20260928 23:14:31.547431 13751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:31.547431 13901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 0 FOLLOWER]: Advancing to term 1
W20260928 23:14:31.547624 13730 tablet.cc:2369] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:31.548254 13751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1de2dee8c1c410ba3fb9a2b8739e28b in term 1.
I20260928 23:14:31.548254 13901 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1de2dee8c1c410ba3fb9a2b8739e28b in term 1.
I20260928 23:14:31.548429 13802 leader_election.cc:302] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631fdb2b723341a5b4f15673f8567454, f1de2dee8c1c410ba3fb9a2b8739e28b; no voters:
I20260928 23:14:31.548507 13969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:31.548645 13969 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [term 1 LEADER]: Becoming Leader. State: Replica: f1de2dee8c1c410ba3fb9a2b8739e28b, State: Running, Role: LEADER
I20260928 23:14:31.548722 13969 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } }
I20260928 23:14:31.549129 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1de2dee8c1c410ba3fb9a2b8739e28b. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.549281 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:31.549491 13984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:31.549672 13984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:31.550854 13751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [term 1 FOLLOWER]: Refusing update from remote peer f1de2dee8c1c410ba3fb9a2b8739e28b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:31.550864 13901 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [term 1 FOLLOWER]: Refusing update from remote peer f1de2dee8c1c410ba3fb9a2b8739e28b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:31.551148 13969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:31.551306 13969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [LEADER]: Connected to new peer: Peer: permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:31.552348 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1de2dee8c1c410ba3fb9a2b8739e28b. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.552335 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1de2dee8c1c410ba3fb9a2b8739e28b. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.552419 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.552419 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.553311 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.553385 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:31.553634 13985 mvcc.cc:205] Tried to move back new op lower bound from 7334450264270233600 to 7334450264264015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334450264270233600}]
I20260928 23:14:31.553714 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.553789 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.554171 13986 mvcc.cc:205] Tried to move back new op lower bound from 7334450264270233600 to 7334450264264015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334450264270233600}]
I20260928 23:14:31.554594 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.554682 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:31.554785 13984 catalog_manager.cc:1347] Generated new cluster ID: 865eda92312c4c7597f77134fa404c12
I20260928 23:14:31.554852 13984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:31.555006 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba8aaa563f642ec9f531116ed2205fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } peers { permanent_uuid: "f1de2dee8c1c410ba3fb9a2b8739e28b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38491 } } peers { permanent_uuid: "631fdb2b723341a5b4f15673f8567454" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44287 } } }
I20260928 23:14:31.555073 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:14:31.555917 13987 mvcc.cc:205] Tried to move back new op lower bound from 7334450264270233600 to 7334450264264015872. Current Snapshot: MvccSnapshot[applied={T|T < 7334450264270233600}]
I20260928 23:14:31.574874 13984 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:14:31.575582 13984 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:31.588760 13984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1de2dee8c1c410ba3fb9a2b8739e28b: Generated new TSK 0
I20260928 23:14:31.589006 13984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:14:32.400058 13949 catalog_manager.cc:1259] Loaded cluster ID: 865eda92312c4c7597f77134fa404c12
I20260928 23:14:32.400135 13949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: loading cluster ID for follower catalog manager: success
I20260928 23:14:32.400913 13949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: acquiring CA information for follower catalog manager: success
I20260928 23:14:32.401161 13949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 631fdb2b723341a5b4f15673f8567454: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:14:32.411401 13966 catalog_manager.cc:1259] Loaded cluster ID: 865eda92312c4c7597f77134fa404c12
I20260928 23:14:32.411468 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: loading cluster ID for follower catalog manager: success
I20260928 23:14:32.412232 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: acquiring CA information for follower catalog manager: success
I20260928 23:14:32.412480 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ba8aaa563f642ec9f531116ed2205fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:14:32.633926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13709
I20260928 23:14:32.638785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13784
I20260928 23:14:32.642933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 13859
2026-09-28T23:14:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2781 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:14:32 dist-test-slave-6vgd krb5kdc[14000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:14:32 dist-test-slave-6vgd krb5kdc[14000](info): set up 2 sockets
Sep 28 23:14:32 dist-test-slave-6vgd krb5kdc[14000](info): commencing operation
krb5kdc: starting...
W20260928 23:14:33.592973 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:14:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:14:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:14:33.700074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:33.778582 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:33.778815 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:33.778838 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:33.779991 14013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:33.780036 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:33.780049 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:33.780061 14013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:33.780072 14013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:33.781571 14013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:33.781807 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:33.782022 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:33.786621 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:33.786765 14013 server_base.cc:1034] running on GCE node
W20260928 23:14:33.786749 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:33.786731 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:33.787113 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:33.787385 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:33.788520 14013 hybrid_clock.cc:648] HybridClock initialized: now 1790637273788503 us; error 34 us; skew 500 ppm
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:33.790761 14013 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:33.791183 14013 webserver.cc:460] Webserver started at http://127.0.98.62:44499/ using document root <none> and password file <none>
I20260928 23:14:33.791381 14013 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:33.791425 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:33.791541 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:33.792541 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "40a088870a144cfb8271a5d114e1014d"
format_stamp: "Formatted at 2026-09-28 23:14:33 on dist-test-slave-6vgd"
server_key: "9dd2c3a1388d6cba3f32b4edf46055c3"
server_key_iv: "7666715c71c3889d693bd5cc81aec138"
server_key_version: "encryptionkey@0"
I20260928 23:14:33.792857 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "40a088870a144cfb8271a5d114e1014d"
format_stamp: "Formatted at 2026-09-28 23:14:33 on dist-test-slave-6vgd"
server_key: "9dd2c3a1388d6cba3f32b4edf46055c3"
server_key_iv: "7666715c71c3889d693bd5cc81aec138"
server_key_version: "encryptionkey@0"
I20260928 23:14:33.794067 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:14:33.794669 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:33.794800 14013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:33.794878 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "40a088870a144cfb8271a5d114e1014d"
format_stamp: "Formatted at 2026-09-28 23:14:33 on dist-test-slave-6vgd"
server_key: "9dd2c3a1388d6cba3f32b4edf46055c3"
server_key_iv: "7666715c71c3889d693bd5cc81aec138"
server_key_version: "encryptionkey@0"
I20260928 23:14:33.794934 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:33.803486 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:33.804997 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:33.805116 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:33.808640 14013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43243
I20260928 23:14:33.808668 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43243 every 8 connection(s)
I20260928 23:14:33.808951 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:33.809703 14081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:33.811729 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d: Bootstrap starting.
I20260928 23:14:33.812204 14081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:33.812415 14081 log.cc:824] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:33.812975 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d: No bootstrap required, opened a new log
I20260928 23:14:33.814041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14013
I20260928 23:14:33.814131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:14:33.814307 392 external_mini_cluster.cc:1428] Setting key b7f8e98b12a7469015189ec7de4a7fe9
I20260928 23:14:33.814556 14081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } }
I20260928 23:14:33.814746 14081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:33.814795 14081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40a088870a144cfb8271a5d114e1014d, State: Initialized, Role: FOLLOWER
I20260928 23:14:33.814961 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } }
I20260928 23:14:33.815044 14081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:33.815073 14081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:33.815115 14081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:33.815979 14081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } }
I20260928 23:14:33.816078 14081 leader_election.cc:302] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40a088870a144cfb8271a5d114e1014d; no voters:
I20260928 23:14:33.816213 14081 leader_election.cc:288] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:14:33.816269 14086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:33.816417 14086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [term 1 LEADER]: Becoming Leader. State: Replica: 40a088870a144cfb8271a5d114e1014d, State: Running, Role: LEADER
I20260928 23:14:33.816437 14081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:33.817169 14086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } }
I20260928 23:14:33.817718 14087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40a088870a144cfb8271a5d114e1014d. Latest consensus state: current_term: 1 leader_uuid: "40a088870a144cfb8271a5d114e1014d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } } }
I20260928 23:14:33.817800 14087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:33.817723 14088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40a088870a144cfb8271a5d114e1014d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40a088870a144cfb8271a5d114e1014d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43243 } } }
I20260928 23:14:33.817844 14088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:33.818017 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:33.819064 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:33.821432 14101 catalog_manager.cc:1347] Generated new cluster ID: d347b5448a6d468c819f3944eddc2759
I20260928 23:14:33.821478 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:33.823899 14085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:33.814721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.98.62:43243)
0928 23:14:33.814933 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:33.814942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:33.814971 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:14:33.815079 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:33.815086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:33.815139 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:33.815278 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:33.815814 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:33.817073 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:33.818150 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:33.818670 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:33.818748 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:33.821093 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:33.821096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:33.821096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:33.821105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:33.822581 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:33.822904 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:33.822907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:33.822907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:33.822994 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:33.823162 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:33.823162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:33.823163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:33.823378 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:33.823525 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:33.823611 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 23:14:33.823749 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
I20260928 23:14:33.824851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:14:33.839236 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:14:33.839924 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:33.844920 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40a088870a144cfb8271a5d114e1014d: Generated new TSK 0
I20260928 23:14:33.845094 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:14:33.878695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14013
2026-09-28T23:14:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1234 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14108](info): set up 2 sockets
Sep 28 23:14:33 dist-test-slave-6vgd krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260928 23:14:34.812973 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:14:34 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:14:34.923897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:35.002667 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.002882 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.002914 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.004088 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:35.004132 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.004149 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.004165 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:35.004179 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:35.005702 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.005945 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.006165 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.010859 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.010895 14121 server_base.cc:1034] running on GCE node
W20260928 23:14:35.010912 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.010867 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.011303 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.011592 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.012735 14121 hybrid_clock.cc:648] HybridClock initialized: now 1790637275012718 us; error 33 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.015059 14121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:35.015494 14121 webserver.cc:460] Webserver started at http://127.0.98.62:44539/ using document root <none> and password file <none>
I20260928 23:14:35.015710 14121 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.015758 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.015879 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:35.016923 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "0eeab012026a4a898514888af1e28f53"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "0827390cc0fdd6f7aa7af1a551cf3d9b"
server_key_iv: "1234cda7e0c2bcbf5a4af6a7d308e83b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.017290 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "0eeab012026a4a898514888af1e28f53"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "0827390cc0fdd6f7aa7af1a551cf3d9b"
server_key_iv: "1234cda7e0c2bcbf5a4af6a7d308e83b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.018518 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:35.019167 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.019316 14121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:35.019395 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "0eeab012026a4a898514888af1e28f53"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "0827390cc0fdd6f7aa7af1a551cf3d9b"
server_key_iv: "1234cda7e0c2bcbf5a4af6a7d308e83b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.019461 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.041949 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.043731 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.043869 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.047642 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35227
I20260928 23:14:35.047684 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35227 every 8 connection(s)
I20260928 23:14:35.048008 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:35.048295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14121
I20260928 23:14:35.048414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:14:35.048659 392 external_mini_cluster.cc:1428] Setting key 220d1326ead7fcdd8050db8f7be517b1
I20260928 23:14:35.049161 14189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:35.052138 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap starting.
I20260928 23:14:35.052798 14189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:35.053066 14189 log.cc:824] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.053725 14189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: No bootstrap required, opened a new log
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:35.055832 14189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.055971 14189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:35.056005 14189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.056172 14189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.056262 14189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:35.056291 14189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:35.056339 14189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:35.056975 14189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.057076 14189 leader_election.cc:302] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eeab012026a4a898514888af1e28f53; no voters:
I20260928 23:14:35.057256 14189 leader_election.cc:288] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:14:35.057302 14194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:35.057451 14194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 LEADER]: Becoming Leader. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Running, Role: LEADER
I20260928 23:14:35.057521 14189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:35.057570 14194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.058058 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:35.058132 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:35.058506 14196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eeab012026a4a898514888af1e28f53. Latest consensus state: current_term: 1 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:35.058583 14196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:35.058699 14203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:35.059187 14203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:35.059456 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.049178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.98.62:35227)
0928 23:14:35.049356 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.049366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.049394 (+ 28us) server_negotiation.cc:407] Connection header received
0928 23:14:35.049833 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.049840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.049887 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.050000 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.050723 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.051597 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.052608 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.053041 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.053119 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.055572 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.055577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.055578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.055589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.057457 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.057834 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.057838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.057838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.057944 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.058164 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.058165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.058165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.058477 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.058626 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.059131 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.059264 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260928 23:14:35.060588 14203 catalog_manager.cc:1347] Generated new cluster ID: d3f4225999554030805d18f2203c5daa
I20260928 23:14:35.060637 14203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:35.090034 14203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:14:35.090785 14203 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:35.099315 14203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Generated new TSK 0
I20260928 23:14:35.099530 14203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:14:35.128388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:35.206780 14215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.207003 14215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.207033 14215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.208189 14215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.208228 14215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.208254 14215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:35.209709 14215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.209933 14215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.210158 14215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.214845 14223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.214864 14220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.214888 14221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.214880 14215 server_base.cc:1034] running on GCE node
I20260928 23:14:35.215245 14215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.215484 14215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.216624 14215 hybrid_clock.cc:648] HybridClock initialized: now 1790637275216610 us; error 30 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.218983 14215 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:35.219434 14215 webserver.cc:460] Webserver started at http://127.0.98.1:32855/ using document root <none> and password file <none>
I20260928 23:14:35.219635 14215 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.219679 14215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.219798 14215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:35.220813 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "209b754129804c90bc7dd663fa6209cb"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "75c2998fecff9cd6e1d77dfef3a84f54"
server_key_iv: "a73f06cedc96274c2acc6bf9bb8ef979"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.221140 14215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "209b754129804c90bc7dd663fa6209cb"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "75c2998fecff9cd6e1d77dfef3a84f54"
server_key_iv: "a73f06cedc96274c2acc6bf9bb8ef979"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.222371 14215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:35.223047 14230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.223224 14215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:35.223312 14215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "209b754129804c90bc7dd663fa6209cb"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "75c2998fecff9cd6e1d77dfef3a84f54"
server_key_iv: "a73f06cedc96274c2acc6bf9bb8ef979"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.223373 14215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.245584 14215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.247296 14215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.247429 14215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.247640 14215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:35.247931 14215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:35.247962 14215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.247991 14215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:35.248014 14215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.253433 14215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34825
I20260928 23:14:35.253491 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34825 every 8 connection(s)
I20260928 23:14:35.253773 14215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:35.261885 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.254401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51555 (local address 127.0.98.62:35227)
0928 23:14:35.254461 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.254463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.254644 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:14:35.255105 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.255106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.255115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.255162 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.256023 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.256517 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.257447 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.257642 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.257689 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.259682 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.259684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.259684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.259690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.260860 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.261167 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.261167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.261167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.261236 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.261421 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.261421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.261422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.261515 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.261567 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.261764 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.261822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:14:35.262051 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.254380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:51555)
0928 23:14:35.254531 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.254568 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.255017 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.255217 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.255220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.255760 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:35.255952 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.255957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.256587 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.256588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.257388 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.257390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.257699 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.257700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.257844 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.258292 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:35.258301 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:35.259600 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:35.260916 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.260919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.260921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.261117 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.261285 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.261286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.261286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.261377 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.261566 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:35.261569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:35.261675 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.261748 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.261856 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260928 23:14:35.262517 14344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:35.262609 14344 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:35.262766 14344 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:35.262825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14215
I20260928 23:14:35.262915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:14:35.263137 392 external_mini_cluster.cc:1428] Setting key 5fe8b3a5c6d5b6fccbfd57d4d982657e
I20260928 23:14:35.263288 14153 ts_manager.cc:194] Registered new tserver with Master: 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825)
I20260928 23:14:35.264078 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51555
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:14:35.285730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:35.363121 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.363309 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.363330 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.364395 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.364431 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.364456 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:35.365890 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.366079 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.366258 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.370873 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.370937 14349 server_base.cc:1034] running on GCE node
W20260928 23:14:35.370873 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.370904 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.371289 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.371541 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.372682 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790637275372666 us; error 29 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.375015 14349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:35.375443 14349 webserver.cc:460] Webserver started at http://127.0.98.2:46705/ using document root <none> and password file <none>
I20260928 23:14:35.375651 14349 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.375698 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.375820 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:35.376801 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "ca12736d41e135b36b959fc742149654"
server_key_iv: "2e1caf411f27583ff3a938d2f99b0c6b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.377125 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "ca12736d41e135b36b959fc742149654"
server_key_iv: "2e1caf411f27583ff3a938d2f99b0c6b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.378322 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:35.378962 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.379117 14349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:35.379194 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "ca12736d41e135b36b959fc742149654"
server_key_iv: "2e1caf411f27583ff3a938d2f99b0c6b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.379257 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.387162 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.388792 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.388916 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.389115 14349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:35.389418 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:35.389451 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.389479 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:35.389503 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.394801 14349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43813
I20260928 23:14:35.394853 14477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43813 every 8 connection(s)
I20260928 23:14:35.395213 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:35.399931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14349
I20260928 23:14:35.400053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:14:35.400261 392 external_mini_cluster.cc:1428] Setting key e03859476bcb1f9941bfb5ed683ebc7e
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:35.403879 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.395867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60631 (local address 127.0.98.62:35227)
0928 23:14:35.395913 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.395915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.396134 (+ 219us) server_negotiation.cc:407] Connection header received
0928 23:14:35.396598 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.396599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.396608 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.396647 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.397560 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.398049 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.399008 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.399186 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.399229 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.401394 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.401396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.401396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.401404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.402627 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.403035 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.403037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.403037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.403093 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.403332 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.403332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.403332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.403427 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.403473 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.403750 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.403815 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:14:35.404121 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.395846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:60631)
0928 23:14:35.396018 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.396058 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.396502 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.396700 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.396703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.397288 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:35.397484 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.397490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.398160 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.398161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.398945 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.398947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.399242 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.399242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.399338 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.399744 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:35.399754 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:35.401307 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:35.402697 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.402704 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.402706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.402964 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.403155 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.403156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.403156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.403280 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.403475 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:35.403479 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:35.403630 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.403729 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.403892 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260928 23:14:35.404587 14478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:35.404675 14478 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:35.404843 14478 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:35.405197 14153 ts_manager.cc:194] Registered new tserver with Master: d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813)
I20260928 23:14:35.405560 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60631
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:14:35.423902 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:35.502018 14483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.502228 14483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.502260 14483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.503358 14483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.503400 14483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.503435 14483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:35.504840 14483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.505057 14483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.505286 14483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.510036 14491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.510031 14489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.510058 14483 server_base.cc:1034] running on GCE node
W20260928 23:14:35.510031 14488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.510416 14483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.510634 14483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.511777 14483 hybrid_clock.cc:648] HybridClock initialized: now 1790637275511763 us; error 29 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.514101 14483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:35.514472 14483 webserver.cc:460] Webserver started at http://127.0.98.3:35139/ using document root <none> and password file <none>
I20260928 23:14:35.514655 14483 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.514703 14483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.514822 14483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:35.515795 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2dd04287b934fd7acf7c93f3226e16b"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "e80619ece53dc7c75ce72cdb3800ed39"
server_key_iv: "88243ed73eddbca24b394a0844bf89ca"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.516110 14483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2dd04287b934fd7acf7c93f3226e16b"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "e80619ece53dc7c75ce72cdb3800ed39"
server_key_iv: "88243ed73eddbca24b394a0844bf89ca"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.517290 14483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:14:35.517942 14498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.518077 14483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:35.518162 14483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "c2dd04287b934fd7acf7c93f3226e16b"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "e80619ece53dc7c75ce72cdb3800ed39"
server_key_iv: "88243ed73eddbca24b394a0844bf89ca"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.518224 14483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.524571 14483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.526193 14483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.526311 14483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.526489 14483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:35.526750 14483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:35.526780 14483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.526805 14483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:35.526829 14483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.532064 14483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40499
I20260928 23:14:35.532110 14611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40499 every 8 connection(s)
I20260928 23:14:35.532380 14483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:35.537976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14483
I20260928 23:14:35.538075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:14:35.538259 392 external_mini_cluster.cc:1428] Setting key c22c33c6cf17eded76cd06f1122ac713
I20260928 23:14:35.540612 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.533005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42333 (local address 127.0.98.62:35227)
0928 23:14:35.533046 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.533048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.533242 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:14:35.533721 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.533723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.533736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.533789 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.534664 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.535146 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.536075 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.536259 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.536312 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.538213 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.538214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.538215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.538222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.539436 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.539778 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.539779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.539780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.539829 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.540044 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.540045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.540045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.540152 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.540205 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.540442 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.540515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:14:35.540753 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.532974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:42333)
0928 23:14:35.533105 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.533140 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.533625 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.533860 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.533864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.534393 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:35.534569 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.534574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.535222 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.535223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.535992 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.535993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.536323 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.536324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.536442 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.536836 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:35.536843 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:35.538138 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:35.539508 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.539514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.539516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.539715 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.539892 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.539892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.539893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.539991 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.540204 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:35.540210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:35.540334 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.540416 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.540556 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260928 23:14:35.541095 14612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:35.541182 14612 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:35.541355 14612 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:35.541671 14153 ts_manager.cc:194] Registered new tserver with Master: c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
I20260928 23:14:35.542004 14153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42333
I20260928 23:14:35.549667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:14:35.555873 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.550889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.0.98.62:35227)
0928 23:14:35.550941 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.550943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.551038 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:14:35.551078 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.551078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.551086 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.551130 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.551587 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.552049 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.552717 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.552897 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.552969 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.553937 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.553938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.553938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.553944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.555109 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.555341 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.555342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.555342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.555385 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.555504 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.555504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.555504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.555621 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.555687 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.555754 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.555806 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:14:35.557106 14153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:14:35.558766 14153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:14:35.564472 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.0.98.2:43813)
0928 23:14:35.560957 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.560961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.561021 (+ 60us) server_negotiation.cc:407] Connection header received
0928 23:14:35.561152 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.561154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.561166 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.561246 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:35.561428 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562238 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.563534 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564113 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564147 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564185 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.564366 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.564399 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260928 23:14:35.564481 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43813 (local address 127.0.0.1:36786)
0928 23:14:35.560929 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.560958 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.561039 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.561270 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.561273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.561275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:35.561334 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.561336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562333 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.562334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.563365 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.563367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564227 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.564277 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564284 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.564335 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.564383 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":45,"threads_started":1}
I20260928 23:14:35.564626 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.1:34825)
0928 23:14:35.561124 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.561129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.561138 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:14:35.561174 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.561176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.561190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.561286 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:35.561469 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562546 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.563483 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564125 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564185 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564233 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.564515 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.564552 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":70,"threads_started":1}
I20260928 23:14:35.564819 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:35090)
0928 23:14:35.560913 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.560954 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.561011 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.561362 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.561363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.561365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:35.561401 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.561402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562651 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.562651 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.563398 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.563400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564265 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.564328 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564332 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.564375 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.564413 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":47,"threads_started":1}
I20260928 23:14:35.564893 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.0.1:35014)
0928 23:14:35.561280 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.561289 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.561341 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.561479 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.561480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.561481 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:35.561622 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.561625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562787 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.562788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.563823 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.563825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564690 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.564755 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564759 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.564809 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.564836 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":29,"threads_started":1}
I20260928 23:14:35.565191 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.560914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.0.98.3:40499)
0928 23:14:35.561096 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.561100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.561336 (+ 236us) server_negotiation.cc:407] Connection header received
0928 23:14:35.561398 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.561400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.561414 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.561481 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:35.561701 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.562695 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.563893 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.564540 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.564608 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.564657 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.564836 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.564876 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260928 23:14:35.565557 14412 tablet_service.cc:1467] Processing CreateTablet for tablet 563882db0ca24a629023f3ed5816800b (DEFAULT_TABLE table=test-table [id=c8a7b58c41374b7ab512663c346a7fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:35.565558 14278 tablet_service.cc:1467] Processing CreateTablet for tablet 563882db0ca24a629023f3ed5816800b (DEFAULT_TABLE table=test-table [id=c8a7b58c41374b7ab512663c346a7fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:35.565723 14546 tablet_service.cc:1467] Processing CreateTablet for tablet 563882db0ca24a629023f3ed5816800b (DEFAULT_TABLE table=test-table [id=c8a7b58c41374b7ab512663c346a7fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:35.565886 14412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563882db0ca24a629023f3ed5816800b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:35.565886 14278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563882db0ca24a629023f3ed5816800b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:35.565966 14546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 563882db0ca24a629023f3ed5816800b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:35.568416 14626 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Bootstrap starting.
I20260928 23:14:35.568893 14627 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Bootstrap starting.
I20260928 23:14:35.568989 14626 tablet_bootstrap.cc:654] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:35.569352 14628 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Bootstrap starting.
I20260928 23:14:35.569561 14627 tablet_bootstrap.cc:654] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:35.569561 14626 log.cc:824] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.569936 14628 tablet_bootstrap.cc:654] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:35.570088 14627 log.cc:824] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.570308 14626 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: No bootstrap required, opened a new log
I20260928 23:14:35.570371 14626 ts_tablet_manager.cc:1397] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:35.570410 14628 log.cc:824] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.570788 14627 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: No bootstrap required, opened a new log
I20260928 23:14:35.570852 14627 ts_tablet_manager.cc:1397] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:14:35.571026 14628 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: No bootstrap required, opened a new log
I20260928 23:14:35.571079 14628 ts_tablet_manager.cc:1397] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:35.572223 14626 raft_consensus.cc:348] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.572393 14626 raft_consensus.cc:374] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:35.572427 14626 raft_consensus.cc:729] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd04287b934fd7acf7c93f3226e16b, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.572540 14626 consensus_queue.cc:260] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.572597 14628 raft_consensus.cc:348] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.572762 14628 raft_consensus.cc:374] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:35.572803 14626 ts_tablet_manager.cc:1428] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:14:35.572803 14628 raft_consensus.cc:729] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cda40f5e3f461aa82e6aa51766ec2f, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.572745 14627 raft_consensus.cc:348] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.572897 14627 raft_consensus.cc:374] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:35.572930 14627 raft_consensus.cc:729] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209b754129804c90bc7dd663fa6209cb, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.572928 14612 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260928 23:14:35.572930 14628 consensus_queue.cc:260] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.573031 14627 consensus_queue.cc:260] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.573233 14478 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260928 23:14:35.573297 14627 ts_tablet_manager.cc:1428] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:14:35.573403 14344 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260928 23:14:35.573746 14628 ts_tablet_manager.cc:1428] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260928 23:14:35.645797 14479 tablet.cc:2369] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:35.648620 14634 raft_consensus.cc:482] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:35.648738 14634 raft_consensus.cc:504] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.649077 14634 leader_election.cc:288] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499), d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813)
I20260928 23:14:35.651727 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.649296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40041 (local address 127.0.98.2:43813)
0928 23:14:35.649336 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.649338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.649375 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:14:35.649460 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.649462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.649473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.649519 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:35.649686 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.650175 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651083 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.651435 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651493 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.651502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.651631 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.651671 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:14:35.651727 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.649263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43813 (local address 127.0.98.1:40041)
0928 23:14:35.649325 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.649337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.649386 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.649526 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.649526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.649528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:35.649593 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.649595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.650247 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.650248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.651011 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.651013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.651501 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.651570 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.651573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.651616 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.651656 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 23:14:35.651889 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.649263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.98.1:34937)
0928 23:14:35.649439 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.649447 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.649507 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.649621 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.649622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.649623 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:35.649714 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.649715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.650320 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.650322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.651155 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.651157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.651665 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.651729 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.651732 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.651775 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.651809 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":18,"thread_start_us":89,"threads_started":1}
I20260928 23:14:35.651883 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34937 (local address 127.0.98.3:40499)
0928 23:14:35.649359 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.649360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.649510 (+ 150us) server_negotiation.cc:407] Connection header received
0928 23:14:35.649556 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.649557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.649569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.649616 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:35.649798 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.650247 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651228 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.651586 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.651650 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.651659 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.651787 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.651823 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:14:35.652202 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563882db0ca24a629023f3ed5816800b" candidate_uuid: "209b754129804c90bc7dd663fa6209cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd04287b934fd7acf7c93f3226e16b" is_pre_election: true
I20260928 23:14:35.652269 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563882db0ca24a629023f3ed5816800b" candidate_uuid: "209b754129804c90bc7dd663fa6209cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" is_pre_election: true
I20260928 23:14:35.652360 14566 raft_consensus.cc:2454] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209b754129804c90bc7dd663fa6209cb in term 0.
I20260928 23:14:35.652384 14432 raft_consensus.cc:2454] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209b754129804c90bc7dd663fa6209cb in term 0.
I20260928 23:14:35.652570 14231 leader_election.cc:302] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb, d2cda40f5e3f461aa82e6aa51766ec2f; no voters:
I20260928 23:14:35.652693 14634 raft_consensus.cc:2781] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:14:35.652758 14634 raft_consensus.cc:482] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:14:35.652773 14634 raft_consensus.cc:3037] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:35.653445 14634 raft_consensus.cc:504] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.653553 14634 leader_election.cc:288] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 1 election: Requested vote from peers c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499), d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813)
I20260928 23:14:35.653684 14566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563882db0ca24a629023f3ed5816800b" candidate_uuid: "209b754129804c90bc7dd663fa6209cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd04287b934fd7acf7c93f3226e16b"
I20260928 23:14:35.653740 14432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "563882db0ca24a629023f3ed5816800b" candidate_uuid: "209b754129804c90bc7dd663fa6209cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
I20260928 23:14:35.653761 14566 raft_consensus.cc:3037] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:35.653805 14432 raft_consensus.cc:3037] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:35.654598 14566 raft_consensus.cc:2454] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209b754129804c90bc7dd663fa6209cb in term 1.
I20260928 23:14:35.654598 14432 raft_consensus.cc:2454] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209b754129804c90bc7dd663fa6209cb in term 1.
I20260928 23:14:35.654768 14231 leader_election.cc:302] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb, d2cda40f5e3f461aa82e6aa51766ec2f; no voters:
I20260928 23:14:35.654847 14634 raft_consensus.cc:2781] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:35.654996 14634 raft_consensus.cc:686] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 LEADER]: Becoming Leader. State: Replica: 209b754129804c90bc7dd663fa6209cb, State: Running, Role: LEADER
I20260928 23:14:35.655087 14634 consensus_queue.cc:237] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.655826 14153 catalog_manager.cc:5495] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 209b754129804c90bc7dd663fa6209cb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "209b754129804c90bc7dd663fa6209cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } health_report { overall_health: UNKNOWN } } }
I20260928 23:14:35.681790 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.679678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.0.98.1:34825)
0928 23:14:35.679737 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.679738 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.679758 (+ 20us) server_negotiation.cc:407] Connection header received
0928 23:14:35.679792 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.679793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.679812 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.679862 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:14:35.679957 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.680420 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.681092 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.681324 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.681370 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.681491 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:14:35.681588 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:14:35.681646 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.681674 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.681692 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:14:35.683493 14566 raft_consensus.cc:1261] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Refusing update from remote peer 209b754129804c90bc7dd663fa6209cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:35.683501 14432 raft_consensus.cc:1261] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Refusing update from remote peer 209b754129804c90bc7dd663fa6209cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:35.683816 14636 consensus_queue.cc:1035] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:35.683950 14634 consensus_queue.cc:1035] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:35.686657 14641 mvcc.cc:205] Tried to move back new op lower bound from 7334450281196847104 to 7334450281083658240. Current Snapshot: MvccSnapshot[applied={T|T < 7334450281196847104 or (T in {7334450281196847104})}]
I20260928 23:14:35.688419 14152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:35.688508 14152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:35.689170 14152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Sending DeleteTablet for 3 replicas of tablet 563882db0ca24a629023f3ed5816800b
I20260928 23:14:35.689523 14278 tablet_service.cc:1514] Processing DeleteTablet for tablet 563882db0ca24a629023f3ed5816800b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35090
I20260928 23:14:35.689560 14412 tablet_service.cc:1514] Processing DeleteTablet for tablet 563882db0ca24a629023f3ed5816800b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36786
I20260928 23:14:35.689568 14546 tablet_service.cc:1514] Processing DeleteTablet for tablet 563882db0ca24a629023f3ed5816800b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35014
I20260928 23:14:35.689864 14649 tablet_replica.cc:330] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: stopping tablet replica
I20260928 23:14:35.689862 14650 tablet_replica.cc:330] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: stopping tablet replica
I20260928 23:14:35.689863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14215
I20260928 23:14:35.690966 14649 raft_consensus.cc:2229] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:35.691084 14649 raft_consensus.cc:2258] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:35.691478 14649 ts_tablet_manager.cc:1905] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:35.693295 14649 ts_tablet_manager.cc:1918] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:35.693367 14649 log.cc:1196] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/563882db0ca24a629023f3ed5816800b
I20260928 23:14:35.693602 14649 ts_tablet_manager.cc:1939] T 563882db0ca24a629023f3ed5816800b P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting consensus metadata
I20260928 23:14:35.693967 14137 catalog_manager.cc:4841] TS d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): tablet 563882db0ca24a629023f3ed5816800b (table test-table [id=c8a7b58c41374b7ab512663c346a7fe1]) successfully deleted
W20260928 23:14:35.694833 14140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34825 (error 108)
W20260928 23:14:35.695860 14621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:14:35.695161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:35112)
0928 23:14:35.695207 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.695218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.695252 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.695827 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 23:14:35.695981 14140 catalog_manager.cc:4584] TS 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 563882db0ca24a629023f3ed5816800b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:14:35.696014 14650 raft_consensus.cc:2229] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:35.696028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14349
I20260928 23:14:35.696123 14650 raft_consensus.cc:2258] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:35.699867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14483
I20260928 23:14:35.703790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14121
I20260928 23:14:35.703852 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.703725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.0.1:35018)
0928 23:14:35.703779 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.703821 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:40499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:14:35.708993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--webserver_interface=127.0.98.62
--webserver_port=44539
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:35.786477 14652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.786676 14652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.786696 14652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.787799 14652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:35.787837 14652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.787848 14652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.787876 14652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:35.787899 14652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:35.789326 14652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.789539 14652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.789742 14652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.797456 14661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.797516 14658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.797533 14652 server_base.cc:1034] running on GCE node
W20260928 23:14:35.797472 14659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.797901 14652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.798223 14652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.799362 14652 hybrid_clock.cc:648] HybridClock initialized: now 1790637275799349 us; error 27 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.801643 14652 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:35.802034 14652 webserver.cc:460] Webserver started at http://127.0.98.62:44539/ using document root <none> and password file <none>
I20260928 23:14:35.802241 14652 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.802299 14652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.803573 14652 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:35.804123 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.804287 14652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.804370 14652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "0eeab012026a4a898514888af1e28f53"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "0827390cc0fdd6f7aa7af1a551cf3d9b"
server_key_iv: "1234cda7e0c2bcbf5a4af6a7d308e83b"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.804747 14652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.820372 14652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.822067 14652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.822199 14652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.825758 14652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35227
I20260928 23:14:35.825794 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35227 every 8 connection(s)
I20260928 23:14:35.826144 14652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:35.827610 14721 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:14:35.828240 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap starting.
I20260928 23:14:35.833047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14652
I20260928 23:14:35.833330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34825
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=32855
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:35.836002 14721 log.cc:824] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.837853 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:35.838078 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap complete.
I20260928 23:14:35.840091 14721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.840317 14721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.840464 14721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.840533 14721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:35.840584 14721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:35.840631 14721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:14:35.841622 14721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.841710 14721 leader_election.cc:302] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eeab012026a4a898514888af1e28f53; no voters:
I20260928 23:14:35.841856 14721 leader_election.cc:288] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:14:35.841899 14725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:14:35.842039 14725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 LEADER]: Becoming Leader. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Running, Role: LEADER
I20260928 23:14:35.842100 14721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:35.842212 14725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:35.842475 14727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:35.842480 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eeab012026a4a898514888af1e28f53. Latest consensus state: current_term: 2 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:35.842563 14727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:35.842571 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:35.842984 14733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:35.843770 14733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:35.843866 14733 catalog_manager.cc:1259] Loaded cluster ID: d3f4225999554030805d18f2203c5daa
I20260928 23:14:35.843899 14733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:35.844129 14733 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 23:14:35.846941 14733 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 23:14:35.850956 14733 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:35.851133 14733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Loaded TSK: 0
I20260928 23:14:35.851356 14733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:14:35.926949 14723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:35.927143 14723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:35.927165 14723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:35.928285 14723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:35.928325 14723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:35.928350 14723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:35.929777 14723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32855
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:35.929980 14723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:35.930181 14723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:35.934787 14748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.934900 14751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:35.934832 14749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:35.935104 14723 server_base.cc:1034] running on GCE node
I20260928 23:14:35.935292 14723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:35.935516 14723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:35.936655 14723 hybrid_clock.cc:648] HybridClock initialized: now 1790637275936646 us; error 32 us; skew 500 ppm
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:35.939014 14723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:35.939435 14723 webserver.cc:460] Webserver started at http://127.0.98.1:32855/ using document root <none> and password file <none>
I20260928 23:14:35.939632 14723 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:35.939698 14723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:35.940925 14723 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:35.941551 14758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.941730 14723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:35.941826 14723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "209b754129804c90bc7dd663fa6209cb"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "75c2998fecff9cd6e1d77dfef3a84f54"
server_key_iv: "a73f06cedc96274c2acc6bf9bb8ef979"
server_key_version: "encryptionkey@0"
I20260928 23:14:35.942243 14723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:35.951346 14723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:35.953676 14723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:35.953819 14723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:35.954042 14723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:35.954479 14765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:14:35.955848 14723 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:14:35.955904 14723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:14:35.955935 14723 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:14:35.956741 14723 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:14:35.956785 14723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:35.956856 14765 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Bootstrap starting.
I20260928 23:14:35.963819 14723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34825
I20260928 23:14:35.963989 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34825 every 8 connection(s)
I20260928 23:14:35.964246 14723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:14:35.966724 14765 log.cc:824] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:35.967890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14723
I20260928 23:14:35.968176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43813
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=46705
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:35.969164 14765 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:35.969450 14765 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Bootstrap complete.
I20260928 23:14:35.970494 14765 ts_tablet_manager.cc:1397] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260928 23:14:35.971742 14765 raft_consensus.cc:348] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.972010 14765 raft_consensus.cc:729] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209b754129804c90bc7dd663fa6209cb, State: Initialized, Role: FOLLOWER
I20260928 23:14:35.972167 14765 consensus_queue.cc:260] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:35.972417 14765 ts_tablet_manager.cc:1428] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:35.976936 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.964926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:59873)
0928 23:14:35.965112 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.965174 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.965856 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.966574 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.966578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.966937 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:35.967192 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.967199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.969236 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.969237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.970681 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.970684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.971484 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.971485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.971619 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.972229 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:35.972239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:35.973877 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:35.975504 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.975517 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.975519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.975723 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.975947 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.975948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.975948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.976045 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.976372 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:35.976378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:35.976502 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.976585 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.976745 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260928 23:14:35.976948 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.965275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59873 (local address 127.0.98.62:35227)
0928 23:14:35.966113 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.966125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.966167 (+ 42us) server_negotiation.cc:407] Connection header received
0928 23:14:35.966301 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.966308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.966367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.966510 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.967319 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.969088 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.970786 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.971364 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.971471 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.973973 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.973977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.973977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.973989 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.975425 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.975801 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.975804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.975805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.975891 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.976106 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.976107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.976107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.976312 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.976379 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.976657 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.976768 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":71,"threads_started":1}
I20260928 23:14:35.977528 14873 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:35.977615 14873 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:35.977768 14873 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:35.978398 14685 ts_manager.cc:194] Registered new tserver with Master: 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825)
I20260928 23:14:35.979406 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59873
I20260928 23:14:35.980244 14873 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
Sep 28 23:14:35 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:14:35.988212 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.978897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:35124)
0928 23:14:35.979070 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:35.979115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:35.979183 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:35.979360 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:35.979364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:35.979696 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:35.979778 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.979781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.980886 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.980887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.981793 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:35.981797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.982384 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.982388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:35.982483 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.983494 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:35.983499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:35.985133 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:35.987070 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.987076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.987078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.987313 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.987509 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:35.987509 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:35.987510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:35.987586 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:35.987864 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:35.987870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:35.987921 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:14:35.987994 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 23:14:35.988106 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260928 23:14:35.988220 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:35.979090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.0.98.1:34825)
0928 23:14:35.979226 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:35.979230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:35.979240 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:14:35.979268 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:35.979269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:35.979283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:35.979348 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:35.979855 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.980822 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.981885 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:35.982300 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:35.982363 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:35.985245 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:35.985253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:35.985253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:35.985260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:35.986983 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.987390 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.987393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.987393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.987453 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:35.987636 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:35.987637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:35.987637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:35.987810 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:35.987881 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:35.988033 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 23:14:35.988127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260928 23:14:35.989120 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 563882db0ca24a629023f3ed5816800b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35124
I20260928 23:14:35.989739 14882 tablet_replica.cc:330] stopping tablet replica
I20260928 23:14:35.989840 14882 raft_consensus.cc:2229] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:35.989907 14882 raft_consensus.cc:2258] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:35.990212 14882 ts_tablet_manager.cc:1905] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:35.992588 14882 ts_tablet_manager.cc:1918] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:35.992668 14882 log.cc:1196] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/563882db0ca24a629023f3ed5816800b
I20260928 23:14:35.992877 14882 ts_tablet_manager.cc:1939] T 563882db0ca24a629023f3ed5816800b P 209b754129804c90bc7dd663fa6209cb: Deleting consensus metadata
I20260928 23:14:35.993374 14672 catalog_manager.cc:4841] TS 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825): tablet 563882db0ca24a629023f3ed5816800b (table test-table [id=c8a7b58c41374b7ab512663c346a7fe1]) successfully deleted
W20260928 23:14:36.057400 14878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.057588 14878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.057608 14878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.058658 14878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.058689 14878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.058713 14878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:36.060062 14878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46705
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.060245 14878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.060426 14878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.064978 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.065089 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.064976 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.065171 14878 server_base.cc:1034] running on GCE node
I20260928 23:14:36.065368 14878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.065554 14878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.066681 14878 hybrid_clock.cc:648] HybridClock initialized: now 1790637276066668 us; error 31 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.074141 14878 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:36.074543 14878 webserver.cc:460] Webserver started at http://127.0.98.2:46705/ using document root <none> and password file <none>
I20260928 23:14:36.074728 14878 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.074779 14878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.076015 14878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.076538 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.076675 14878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.076752 14878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "ca12736d41e135b36b959fc742149654"
server_key_iv: "2e1caf411f27583ff3a938d2f99b0c6b"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.077064 14878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.089680 14878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.091653 14878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.091781 14878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.091979 14878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:36.092262 14878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:36.092293 14878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.092321 14878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:36.092344 14878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.097777 14878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43813
I20260928 23:14:36.097826 15010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43813 every 8 connection(s)
I20260928 23:14:36.098133 14878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:36.099752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 14878
I20260928 23:14:36.100006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40499
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=35139
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:36.108103 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.098784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38339 (local address 127.0.98.62:35227)
0928 23:14:36.098840 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.098842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.099048 (+ 206us) server_negotiation.cc:407] Connection header received
0928 23:14:36.099511 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.099513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.099524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.099575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:36.100588 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.101363 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.102765 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.102960 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.103004 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.105414 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:36.105416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:36.105416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:36.105423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:36.106689 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.107131 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.107132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.107133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.107179 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.107450 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.107451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.107451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.107556 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:36.107606 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.107950 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.108034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:14:36.108338 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.098779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:38339)
0928 23:14:36.098937 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.098973 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.099432 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.099653 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.099657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.100294 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:36.100493 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.100499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.101460 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.101461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.102677 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.102680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.103027 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.103028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.103180 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.103704 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:36.103715 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:36.105324 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:36.106763 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.106770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.106773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.107065 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.107232 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.107233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.107233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.107394 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.107614 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:36.107621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:36.107800 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.107923 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.108096 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260928 23:14:36.108807 15011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:36.108896 15011 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:36.109059 15011 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:36.109447 14685 ts_manager.cc:194] Registered new tserver with Master: d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813)
I20260928 23:14:36.109808 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38339
W20260928 23:14:36.180315 15014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.180570 15014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.180601 15014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.181834 15014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.181874 15014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.181908 15014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:36.183346 15014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35139
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.183574 15014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.183784 15014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.188673 15022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.188715 15019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.188753 15014 server_base.cc:1034] running on GCE node
W20260928 23:14:36.188689 15020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.189093 15014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.189365 15014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.190523 15014 hybrid_clock.cc:648] HybridClock initialized: now 1790637276190512 us; error 36 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.192917 15014 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:36.193363 15014 webserver.cc:460] Webserver started at http://127.0.98.3:35139/ using document root <none> and password file <none>
I20260928 23:14:36.193558 15014 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.193624 15014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.195012 15014 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:36.195597 15029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.195737 15014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:36.195814 15014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "c2dd04287b934fd7acf7c93f3226e16b"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "e80619ece53dc7c75ce72cdb3800ed39"
server_key_iv: "88243ed73eddbca24b394a0844bf89ca"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.196144 15014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.203382 15014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.205312 15014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.205436 15014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.205626 15014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:36.205991 15036 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:14:36.207329 15014 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:14:36.207367 15014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:36.207397 15014 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:14:36.208019 15014 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:14:36.208048 15014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:36.208098 15036 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Bootstrap starting.
I20260928 23:14:36.209591 15036 log.cc:824] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:36.212369 15036 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:36.212651 15036 tablet_bootstrap.cc:492] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Bootstrap complete.
I20260928 23:14:36.212790 15036 ts_tablet_manager.cc:1397] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 23:14:36.213713 15014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40499
I20260928 23:14:36.213762 15144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40499 every 8 connection(s)
I20260928 23:14:36.214041 15014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:14:36.214747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15014
I20260928 23:14:36.214344 15036 raft_consensus.cc:348] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:36.215197 15036 raft_consensus.cc:729] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd04287b934fd7acf7c93f3226e16b, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.215361 15036 consensus_queue.cc:260] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } } peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } }
I20260928 23:14:36.215627 15036 ts_tablet_manager.cc:1428] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:36.221174 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.215083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.98.62:35227)
0928 23:14:36.215217 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.215219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.215342 (+ 123us) server_negotiation.cc:407] Connection header received
0928 23:14:36.215404 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.215405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.215415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.215473 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:36.216131 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.216600 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.217350 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.217643 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.217713 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.218865 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:36.218866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:36.218867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:36.218873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:36.220163 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.220452 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.220453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.220453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.220513 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.220683 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.220683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.220683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.220774 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:36.220925 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.220963 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.221072 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260928 23:14:36.223052 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.214646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59673 (local address 127.0.98.62:35227)
0928 23:14:36.214691 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.214693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.214867 (+ 174us) server_negotiation.cc:407] Connection header received
0928 23:14:36.215165 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.215167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.215178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.215209 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:36.216056 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.216576 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.217733 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.217978 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.218027 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.220533 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:36.220534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:36.220534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:36.220540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:36.221997 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.222326 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.222326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.222327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.222396 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.222606 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.222606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.222606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.222694 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:36.222742 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.222938 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.222994 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:14:36.223277 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.214613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:59673)
0928 23:14:36.214752 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.214789 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.215058 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.215387 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.215392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.215761 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:36.215961 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.215966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.216678 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.216679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.217665 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.217667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.218046 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.218047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.218181 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.218579 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:36.218594 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:36.220452 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:36.222063 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.222068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.222070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.222270 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.222459 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.222460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.222461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.222557 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.222741 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:36.222744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:36.222848 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.222922 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.223038 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260928 23:14:36.223654 15145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35227
I20260928 23:14:36.223750 15145 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:36.223908 15145 heartbeater.cc:511] Master 127.0.98.62:35227 requested a full tablet report, sending...
I20260928 23:14:36.224362 14685 ts_manager.cc:194] Registered new tserver with Master: c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
I20260928 23:14:36.224821 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59673
I20260928 23:14:36.225513 15145 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 23:14:36.230871 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.224620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.0.1:35032)
0928 23:14:36.224668 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.224678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.224714 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.225245 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.225246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.225364 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:36.225402 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.225403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.226315 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.226317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.226859 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.226861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.227337 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.227340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.227407 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.227718 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:36.227721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:36.228792 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:36.230147 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.230148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.230148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.230296 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.230438 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:36.230439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:36.230439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:36.230481 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:36.230663 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:36.230663 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:36.230693 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.230742 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.230810 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 23:14:36.230885 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.224934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.98.3:40499)
0928 23:14:36.225083 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.225087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.225097 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:14:36.225127 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.225128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.225139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.225200 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:36.225450 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.226247 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.226927 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.227259 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.227327 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.228866 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:36.228873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:36.228873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:36.228878 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:36.230085 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.230351 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.230353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.230353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.230398 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:36.230512 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:36.230513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:36.230513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:36.230623 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:36.230675 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.230763 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.230818 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260928 23:14:36.231269 15079 tablet_service.cc:1514] Processing DeleteTablet for tablet 563882db0ca24a629023f3ed5816800b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35032
I20260928 23:14:36.231781 15152 tablet_replica.cc:330] stopping tablet replica
I20260928 23:14:36.231864 15152 raft_consensus.cc:2229] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:36.231912 15152 raft_consensus.cc:2258] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:36.232194 15152 ts_tablet_manager.cc:1905] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:36.232703 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:14:36.234613 15152 ts_tablet_manager.cc:1918] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:36.234684 15152 log.cc:1196] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/563882db0ca24a629023f3ed5816800b
I20260928 23:14:36.234865 15152 ts_tablet_manager.cc:1939] T 563882db0ca24a629023f3ed5816800b P c2dd04287b934fd7acf7c93f3226e16b: Deleting consensus metadata
I20260928 23:14:36.235257 14672 catalog_manager.cc:4841] TS c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): tablet 563882db0ca24a629023f3ed5816800b (table test-table [id=c8a7b58c41374b7ab512663c346a7fe1]) successfully deleted
I20260928 23:14:36.235438 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.233016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.0.98.62:35227)
0928 23:14:36.233052 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.233054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.233107 (+ 53us) server_negotiation.cc:407] Connection header received
0928 23:14:36.233182 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.233182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.233228 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.233293 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:14:36.233406 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.233843 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.234710 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.234954 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.235007 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.235159 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:14:36.235250 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:14:36.235324 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.235362 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.235378 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:14:36.235666 14685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:14:36.237138 14685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:14:36.238946 15079 tablet_service.cc:1467] Processing CreateTablet for tablet 150d66abd54c433c8fd07158c64efda5 (DEFAULT_TABLE table=test-table [id=e7ba7a2eef3a411fb30db992270c3e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:36.238991 14807 tablet_service.cc:1467] Processing CreateTablet for tablet 150d66abd54c433c8fd07158c64efda5 (DEFAULT_TABLE table=test-table [id=e7ba7a2eef3a411fb30db992270c3e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:36.239092 15079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150d66abd54c433c8fd07158c64efda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:36.239133 14807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150d66abd54c433c8fd07158c64efda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:36.240474 15036 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Bootstrap starting.
I20260928 23:14:36.240494 14765 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Bootstrap starting.
I20260928 23:14:36.240729 14765 tablet_bootstrap.cc:654] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:36.240860 15036 tablet_bootstrap.cc:654] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:36.241295 14765 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: No bootstrap required, opened a new log
I20260928 23:14:36.241343 14765 ts_tablet_manager.cc:1397] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:36.241432 15036 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: No bootstrap required, opened a new log
I20260928 23:14:36.241482 15036 ts_tablet_manager.cc:1397] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.241698 14765 raft_consensus.cc:348] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.241770 14765 raft_consensus.cc:374] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:36.241770 15036 raft_consensus.cc:348] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.241813 14765 raft_consensus.cc:729] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209b754129804c90bc7dd663fa6209cb, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.241834 15036 raft_consensus.cc:374] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:36.241876 15036 raft_consensus.cc:729] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2dd04287b934fd7acf7c93f3226e16b, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.241859 14765 consensus_queue.cc:260] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.241919 15036 consensus_queue.cc:260] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.241940 14765 ts_tablet_manager.cc:1428] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.242017 15036 ts_tablet_manager.cc:1428] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:36.242377 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.238750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43813 (local address 127.0.0.1:36796)
0928 23:14:36.238790 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.238799 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.238831 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.239166 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.239167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.239169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:36.239202 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.239203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.240139 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.240140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.241084 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.241087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.242155 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.242157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.242226 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.242247 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.242279 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.242319 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:14:36.242729 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.238855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.0.98.2:43813)
0928 23:14:36.239022 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.239026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.239036 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:14:36.239069 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.239071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.239083 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.239152 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:36.239267 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.240059 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.241179 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.242062 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.242521 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.242588 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.242614 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.242651 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260928 23:14:36.243309 14945 tablet_service.cc:1467] Processing CreateTablet for tablet 150d66abd54c433c8fd07158c64efda5 (DEFAULT_TABLE table=test-table [id=e7ba7a2eef3a411fb30db992270c3e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:36.243584 14945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150d66abd54c433c8fd07158c64efda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:36.246176 15157 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Bootstrap starting.
I20260928 23:14:36.246706 15157 tablet_bootstrap.cc:654] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:36.247200 15157 log.cc:824] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:36.247843 15157 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: No bootstrap required, opened a new log
I20260928 23:14:36.247900 15157 ts_tablet_manager.cc:1397] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:36.249150 15157 raft_consensus.cc:348] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.249330 15157 raft_consensus.cc:374] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:36.249375 15157 raft_consensus.cc:729] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cda40f5e3f461aa82e6aa51766ec2f, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.249482 15157 consensus_queue.cc:260] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.249701 15157 ts_tablet_manager.cc:1428] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:14:36.249739 15011 heartbeater.cc:503] Master 127.0.98.62:35227 was elected leader, sending a full tablet report...
I20260928 23:14:36.322961 15159 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:36.323100 15159 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.323447 15159 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
I20260928 23:14:36.326424 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.323677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32937 (local address 127.0.98.1:34825)
0928 23:14:36.323730 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.323732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.323740 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:36.323825 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.323826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.323853 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.323931 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:36.324092 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.324666 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.325719 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.326101 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.326153 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.326224 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.326315 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.326355 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:14:36.326439 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.323870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46585 (local address 127.0.98.3:40499)
0928 23:14:36.323923 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.323925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.323952 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:14:36.324018 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.324019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.324041 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.324102 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:36.324245 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.324695 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.325706 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.326123 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.326190 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.326258 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.326345 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.326383 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:14:36.326431 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.323617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.98.2:32937)
0928 23:14:36.323672 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.323684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.323726 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.323950 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.323952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.323954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:36.324025 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.324026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.324755 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.324756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.325628 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.325630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.326171 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.326172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.326252 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.326255 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.326304 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.326342 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":43}
I20260928 23:14:36.326442 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.323737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.98.2:46585)
0928 23:14:36.323901 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.323911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.323966 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.324109 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:36.324110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:36.324112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:36.324172 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.324174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.324767 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.324768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.325629 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:36.325631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.326210 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.326211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:36.326273 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.326276 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:36.326325 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 23:14:36.326359 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":29,"thread_start_us":90,"threads_started":1}
I20260928 23:14:36.326785 14822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150d66abd54c433c8fd07158c64efda5" candidate_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209b754129804c90bc7dd663fa6209cb" is_pre_election: true
I20260928 23:14:36.326797 15099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150d66abd54c433c8fd07158c64efda5" candidate_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd04287b934fd7acf7c93f3226e16b" is_pre_election: true
I20260928 23:14:36.326921 15099 raft_consensus.cc:2454] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2cda40f5e3f461aa82e6aa51766ec2f in term 0.
I20260928 23:14:36.326923 14822 raft_consensus.cc:2454] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2cda40f5e3f461aa82e6aa51766ec2f in term 0.
I20260928 23:14:36.327113 14901 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f; no voters:
I20260928 23:14:36.327229 15159 raft_consensus.cc:2781] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:14:36.327280 15159 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:14:36.327296 15159 raft_consensus.cc:3037] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:36.327952 15159 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.328053 15159 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [CANDIDATE]: Term 1 election: Requested vote from peers 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
I20260928 23:14:36.328214 15099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150d66abd54c433c8fd07158c64efda5" candidate_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2dd04287b934fd7acf7c93f3226e16b"
I20260928 23:14:36.328215 14822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150d66abd54c433c8fd07158c64efda5" candidate_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209b754129804c90bc7dd663fa6209cb"
I20260928 23:14:36.328277 15099 raft_consensus.cc:3037] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:36.328284 14822 raft_consensus.cc:3037] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:36.328866 15099 raft_consensus.cc:2454] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2cda40f5e3f461aa82e6aa51766ec2f in term 1.
I20260928 23:14:36.328989 14822 raft_consensus.cc:2454] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2cda40f5e3f461aa82e6aa51766ec2f in term 1.
I20260928 23:14:36.329013 14901 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f; no voters:
I20260928 23:14:36.329079 15159 raft_consensus.cc:2781] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:36.329259 15159 raft_consensus.cc:686] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 LEADER]: Becoming Leader. State: Replica: d2cda40f5e3f461aa82e6aa51766ec2f, State: Running, Role: LEADER
I20260928 23:14:36.329348 15159 consensus_queue.cc:237] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.330142 14684 catalog_manager.cc:5495] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f reported cstate change: term changed from 0 to 1, leader changed from <none> to d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } health_report { overall_health: UNKNOWN } } }
W20260928 23:14:36.348628 15012 tablet.cc:2369] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:36.357548 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.355485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.0.98.2:43813)
0928 23:14:36.355536 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:36.355538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:36.355545 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:14:36.355578 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:36.355579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:36.355596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:36.355664 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:14:36.355757 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.356243 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.356898 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:36.357094 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:36.357139 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:36.357272 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:14:36.357368 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:14:36.357425 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:36.357440 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 23:14:36.357457 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:14:36.359237 14822 raft_consensus.cc:1261] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Refusing update from remote peer d2cda40f5e3f461aa82e6aa51766ec2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:36.359272 15099 raft_consensus.cc:1261] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Refusing update from remote peer d2cda40f5e3f461aa82e6aa51766ec2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:36.359555 15161 consensus_queue.cc:1035] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:36.359683 15159 consensus_queue.cc:1035] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:36.361982 15165 mvcc.cc:205] Tried to move back new op lower bound from 7334450283964665856 to 7334450283845496832. Current Snapshot: MvccSnapshot[applied={T|T < 7334450283964665856}]
I20260928 23:14:36.362309 15166 mvcc.cc:205] Tried to move back new op lower bound from 7334450283964665856 to 7334450283845496832. Current Snapshot: MvccSnapshot[applied={T|T < 7334450283964665856}]
I20260928 23:14:36.363765 14684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:36.363842 14684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:36.364475 14684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Sending DeleteTablet for 3 replicas of tablet 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:36.364786 14807 tablet_service.cc:1514] Processing DeleteTablet for tablet 150d66abd54c433c8fd07158c64efda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35124
I20260928 23:14:36.364840 15079 tablet_service.cc:1514] Processing DeleteTablet for tablet 150d66abd54c433c8fd07158c64efda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35032
I20260928 23:14:36.364874 14882 tablet_replica.cc:330] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: stopping tablet replica
I20260928 23:14:36.364912 15152 tablet_replica.cc:330] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: stopping tablet replica
I20260928 23:14:36.364915 14882 raft_consensus.cc:2229] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:36.364950 15152 raft_consensus.cc:2229] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:36.364966 14882 raft_consensus.cc:2258] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:36.364985 15152 raft_consensus.cc:2258] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:36.365083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14723
I20260928 23:14:36.365355 15152 ts_tablet_manager.cc:1905] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:36.366885 15152 ts_tablet_manager.cc:1918] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:36.366936 15152 log.cc:1196] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/150d66abd54c433c8fd07158c64efda5
I20260928 23:14:36.367146 15152 ts_tablet_manager.cc:1939] T 150d66abd54c433c8fd07158c64efda5 P c2dd04287b934fd7acf7c93f3226e16b: Deleting consensus metadata
I20260928 23:14:36.367429 14672 catalog_manager.cc:4841] TS c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): tablet 150d66abd54c433c8fd07158c64efda5 (table test-table [id=e7ba7a2eef3a411fb30db992270c3e12]) successfully deleted
I20260928 23:14:36.368046 14945 tablet_service.cc:1514] Processing DeleteTablet for tablet 150d66abd54c433c8fd07158c64efda5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36796
I20260928 23:14:36.368265 15173 tablet_replica.cc:330] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: stopping tablet replica
I20260928 23:14:36.368371 15173 raft_consensus.cc:2229] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:14:36.368463 15173 raft_consensus.cc:2258] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:36.368899 15173 ts_tablet_manager.cc:1905] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:36.370469 15173 ts_tablet_manager.cc:1918] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:36.370559 15173 log.cc:1196] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/150d66abd54c433c8fd07158c64efda5
I20260928 23:14:36.370842 15173 ts_tablet_manager.cc:1939] T 150d66abd54c433c8fd07158c64efda5 P d2cda40f5e3f461aa82e6aa51766ec2f: Deleting consensus metadata
W20260928 23:14:36.371132 14672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34825 (error 108)
I20260928 23:14:36.371340 14669 catalog_manager.cc:4841] TS d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): tablet 150d66abd54c433c8fd07158c64efda5 (table test-table [id=e7ba7a2eef3a411fb30db992270c3e12]) successfully deleted
I20260928 23:14:36.371377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14878
I20260928 23:14:36.372809 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.372675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34825 (local address 127.0.0.1:35138)
0928 23:14:36.372738 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.372778 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 23:14:36.372954 14672 catalog_manager.cc:4584] TS 209b754129804c90bc7dd663fa6209cb (127.0.98.1:34825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 150d66abd54c433c8fd07158c64efda5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34825: connect: Connection refused (error 111)
I20260928 23:14:36.375556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15014
I20260928 23:14:36.379335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 14652
I20260928 23:14:36.383877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--webserver_interface=127.0.98.62
--webserver_port=44539
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:36.462018 15175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.462234 15175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.462263 15175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.463382 15175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:36.463419 15175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.463438 15175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.463452 15175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:36.463467 15175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:36.464874 15175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.465111 15175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.465332 15175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.469759 15183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.469858 15181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.469794 15180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.469962 15175 server_base.cc:1034] running on GCE node
I20260928 23:14:36.470227 15175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.470495 15175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.471645 15175 hybrid_clock.cc:648] HybridClock initialized: now 1790637276471631 us; error 28 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.473950 15175 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:36.474340 15175 webserver.cc:460] Webserver started at http://127.0.98.62:44539/ using document root <none> and password file <none>
I20260928 23:14:36.474555 15175 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.474603 15175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.475855 15175 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:36.476490 15190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.476676 15175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:14:36.476776 15175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "0eeab012026a4a898514888af1e28f53"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "0827390cc0fdd6f7aa7af1a551cf3d9b"
server_key_iv: "1234cda7e0c2bcbf5a4af6a7d308e83b"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.477094 15175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.497625 15175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.499358 15175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.499488 15175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.503021 15175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35227
I20260928 23:14:36.503062 15242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35227 every 8 connection(s)
I20260928 23:14:36.503384 15175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:36.504858 15243 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:14:36.505486 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap starting.
I20260928 23:14:36.508111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15175
I20260928 23:14:36.508409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34825
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=32855
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:36.513760 15243 log.cc:824] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:36.515857 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:36.516124 15243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53: Bootstrap complete.
I20260928 23:14:36.518219 15243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:36.518438 15243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.518576 15243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:36.518633 15243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:36.518673 15243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:36.518718 15243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:14:36.519650 15243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:36.519738 15243 leader_election.cc:302] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eeab012026a4a898514888af1e28f53; no voters:
I20260928 23:14:36.519886 15243 leader_election.cc:288] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:14:36.519932 15247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:14:36.520047 15247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [term 3 LEADER]: Becoming Leader. State: Replica: 0eeab012026a4a898514888af1e28f53, State: Running, Role: LEADER
I20260928 23:14:36.520138 15243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:36.520126 15247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260928 23:14:36.520416 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eeab012026a4a898514888af1e28f53. Latest consensus state: current_term: 3 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:36.520509 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:36.520414 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0eeab012026a4a898514888af1e28f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eeab012026a4a898514888af1e28f53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260928 23:14:36.520815 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eeab012026a4a898514888af1e28f53 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:36.520944 15253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:36.521711 15253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:36.521816 15253 catalog_manager.cc:1259] Loaded cluster ID: d3f4225999554030805d18f2203c5daa
I20260928 23:14:36.521847 15253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 23:14:36.522205 15253 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdce48479ad google::LogMessage::Fail() at ??:0
@ 0x7fdce484a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdce48474de google::LogMessage::Flush() at ??:0
@ 0x7fdce4848a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdcec7680f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdcec7718b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdce67150d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdce670a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdce756e6db start_thread at ??:0
@ 0x7fdce334561f clone at ??:0
W20260928 23:14:36.601502 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.601692 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.601715 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.602885 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.602923 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.602947 15245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:36.604362 15245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32855
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.604566 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.604784 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.609588 15279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.609705 15245 server_base.cc:1034] running on GCE node
W20260928 23:14:36.609803 15282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.609724 15280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.610083 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.610308 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.611449 15245 hybrid_clock.cc:648] HybridClock initialized: now 1790637276611435 us; error 28 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.613965 15245 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:36.614403 15245 webserver.cc:460] Webserver started at http://127.0.98.1:32855/ using document root <none> and password file <none>
I20260928 23:14:36.614611 15245 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.614657 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.616024 15245 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.616611 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.616778 15245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:36.616861 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "209b754129804c90bc7dd663fa6209cb"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "75c2998fecff9cd6e1d77dfef3a84f54"
server_key_iv: "a73f06cedc96274c2acc6bf9bb8ef979"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.617198 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.664582 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.666941 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.667094 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.667368 15245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:36.667871 15296 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:14:36.669298 15245 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:14:36.669356 15245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:14:36.669389 15245 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:14:36.670245 15245 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:14:36.670288 15245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:14:36.670342 15296 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Bootstrap starting.
I20260928 23:14:36.677032 15245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34825
I20260928 23:14:36.677243 15403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34825 every 8 connection(s)
I20260928 23:14:36.677472 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:14:36.680018 15296 log.cc:824] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:36.682039 15296 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:14:36.682268 15296 tablet_bootstrap.cc:492] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Bootstrap complete.
I20260928 23:14:36.682981 15296 ts_tablet_manager.cc:1397] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260928 23:14:36.683800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15245
I20260928 23:14:36.684089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43813
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=46705
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:14:36.684121 15296 raft_consensus.cc:348] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.684422 15296 raft_consensus.cc:729] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209b754129804c90bc7dd663fa6209cb, State: Initialized, Role: FOLLOWER
I20260928 23:14:36.684556 15296 consensus_queue.cc:260] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:36.684813 15296 ts_tablet_manager.cc:1428] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260928 23:14:36.725343 15406 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:14:36.678205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:37529)
0928 23:14:36.678410 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.678475 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:36.679125 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:36.725135 (+ 46010us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
W20260928 23:14:36.725548 15404 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35227 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35227: Client connection negotiation failed: client connection to 127.0.98.62:35227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:14:36.725848 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.725744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:59503)
0928 23:14:36.725787 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.725826 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:14:36.726166 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.726083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:41569)
0928 23:14:36.726123 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.726145 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:14:36.726282 15404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:14:36.768805 15409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.769013 15409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.769037 15409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.770229 15409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.770274 15409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.770313 15409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:36.771855 15409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46705
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.772068 15409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.772295 15409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.777143 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.777318 15409 server_base.cc:1034] running on GCE node
W20260928 23:14:36.777513 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.777688 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.777937 15409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.778254 15409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.779407 15409 hybrid_clock.cc:648] HybridClock initialized: now 1790637276779393 us; error 30 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.781801 15409 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:36.782209 15409 webserver.cc:460] Webserver started at http://127.0.98.2:46705/ using document root <none> and password file <none>
I20260928 23:14:36.782395 15409 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.782433 15409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.783624 15409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.784101 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.784240 15409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.784328 15409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "d2cda40f5e3f461aa82e6aa51766ec2f"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "ca12736d41e135b36b959fc742149654"
server_key_iv: "2e1caf411f27583ff3a938d2f99b0c6b"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.784659 15409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.795531 15409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.797662 15409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.797804 15409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.798022 15409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:36.798310 15409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:36.798341 15409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.798370 15409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:36.798391 15409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.803730 15409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43813
I20260928 23:14:36.803771 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43813 every 8 connection(s)
I20260928 23:14:36.804116 15409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:36.805100 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.804740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:51905)
0928 23:14:36.804890 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.804959 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260928 23:14:36.805485 15538 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35227 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35227: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
I20260928 23:14:36.805724 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.805645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:59353)
0928 23:14:36.805684 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.805707 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:14:36.805922 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.805873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:37313)
0928 23:14:36.805893 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.805910 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 23:14:36.805992 15538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:14:36.808907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15409
I20260928 23:14:36.809140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40499
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35227
--webserver_port=35139
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:36.887234 15541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:36.887434 15541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:36.887455 15541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:36.888618 15541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:36.888660 15541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:36.888689 15541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:36.890118 15541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35139
--tserver_master_addrs=127.0.98.62:35227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:36.890337 15541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:36.890543 15541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:36.895146 15547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.895229 15546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:36.895159 15549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:36.895385 15541 server_base.cc:1034] running on GCE node
I20260928 23:14:36.895613 15541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:36.895813 15541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:36.896944 15541 hybrid_clock.cc:648] HybridClock initialized: now 1790637276896933 us; error 28 us; skew 500 ppm
Sep 28 23:14:36 dist-test-slave-6vgd krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:36.899240 15541 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:36.899676 15541 webserver.cc:460] Webserver started at http://127.0.98.3:35139/ using document root <none> and password file <none>
I20260928 23:14:36.899874 15541 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:36.899921 15541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:36.901149 15541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:14:36.901710 15556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.901854 15541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.901932 15541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "c2dd04287b934fd7acf7c93f3226e16b"
format_stamp: "Formatted at 2026-09-28 23:14:35 on dist-test-slave-6vgd"
server_key: "e80619ece53dc7c75ce72cdb3800ed39"
server_key_iv: "88243ed73eddbca24b394a0844bf89ca"
server_key_version: "encryptionkey@0"
I20260928 23:14:36.902261 15541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:36.930439 15541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:36.932356 15541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:36.932497 15541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:36.932716 15541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:36.933012 15541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:36.933050 15541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.933079 15541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:36.933100 15541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:36.938551 15541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40499
I20260928 23:14:36.938607 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40499 every 8 connection(s)
I20260928 23:14:36.938920 15541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:14:36.939893 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.939525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:39339)
0928 23:14:36.939664 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.939737 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
W20260928 23:14:36.940261 15670 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35227 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35227: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
I20260928 23:14:36.940518 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.940436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:43101)
0928 23:14:36.940477 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.940501 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:14:36.940752 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:36.940679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:49443)
0928 23:14:36.940712 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:36.940734 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:14:36.940845 15670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:14:36.943842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15541
I20260928 23:14:37.726783 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:37.726558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:38829)
0928 23:14:37.726683 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:37.726715 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260928 23:14:37.806417 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:37.806234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:56097)
0928 23:14:37.806333 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:37.806364 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260928 23:14:37.941354 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:37.941139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:47787)
0928 23:14:37.941273 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:37.941304 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":53,"threads_started":1}
I20260928 23:14:38.080017 15677 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:38.080143 15677 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:38.080483 15677 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
Sep 28 23:14:38 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 23:14:38 dist-test-slave-6vgd krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637276, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 23:14:38.090512 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.080789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47023 (local address 127.0.98.2:43813)
0928 23:14:38.080938 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:38.080945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:38.080971 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:14:38.081512 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:38.081515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:38.081577 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:38.081721 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:38.082434 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.083274 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084255 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.084656 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:38.086999 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:38.087003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:38.087004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:38.087015 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:38.088924 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089401 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:38.089404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:38.089405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:38.089494 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089728 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:38.089729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:38.089730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:38.089951 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:38.090022 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:38.090284 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 23:14:38.090405 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260928 23:14:38.090515 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.080699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45289 (local address 127.0.98.3:40499)
0928 23:14:38.080872 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:38.080884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:38.080917 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:14:38.081515 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:38.081518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:38.081577 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:38.081721 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:38.082435 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.083260 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084248 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.084673 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084778 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:38.087430 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:38.087433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:38.087434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:38.087443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:38.089001 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089398 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:38.089401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:38.089401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:38.089491 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089725 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:38.089727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:38.089727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:38.089946 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:38.090015 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:38.090284 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 23:14:38.090397 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260928 23:14:38.090533 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.080625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43813 (local address 127.0.98.1:47023)
0928 23:14:38.080749 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:38.080759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:38.080811 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:38.081838 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:38.081839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:38.082114 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:38.082315 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:38.082321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.083369 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.083370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:38.084167 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:38.084169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.084742 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:38.084852 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:38.085428 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:38.085436 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:38.086894 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:38.089005 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:38.089013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:38.089328 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:38.089555 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:38.089556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:38.089671 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:38.090016 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:38.090017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:38.090138 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 23:14:38.090224 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:14:38.090402 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":66,"thread_start_us":55,"threads_started":1}
I20260928 23:14:38.090788 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.080621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40499 (local address 127.0.98.1:45289)
0928 23:14:38.080733 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:38.080744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:38.080800 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:38.081767 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:38.081770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:38.082123 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:38.082315 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:38.082322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.083358 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.083359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:38.084166 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:38.084169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:38.084755 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:38.084756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:38.084851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:38.085463 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:38.085467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:38.087324 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:38.089074 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:38.089075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:38.089327 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:38.089550 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:38.089550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:38.089551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:38.089669 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:38.090010 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:38.090016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:38.090138 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 23:14:38.090224 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:14:38.090472 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":170}
W20260928 23:14:38.091454 15290 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
W20260928 23:14:38.091648 15293 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:38.091704 15293 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb; no voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f
I20260928 23:14:38.091787 15677 raft_consensus.cc:2726] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:14:38.727368 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.727170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:57425)
0928 23:14:38.727288 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:38.727324 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260928 23:14:38.806952 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.806770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:43919)
0928 23:14:38.806878 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:38.806912 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260928 23:14:38.941956 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:38.941737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:60523)
0928 23:14:38.941880 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:38.941916 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260928 23:14:39.728003 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:39.727805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:43477)
0928 23:14:39.727917 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:39.727956 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260928 23:14:39.796741 15685 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:39.796835 15685 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:39.796994 15685 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
W20260928 23:14:39.797399 15293 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
W20260928 23:14:39.797499 15290 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:39.797595 15290 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb; no voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f
I20260928 23:14:39.797663 15685 raft_consensus.cc:2726] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:14:39.807587 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:39.807365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:36017)
0928 23:14:39.807502 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:39.807544 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260928 23:14:39.942487 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:39.942317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:56603)
0928 23:14:39.942420 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:39.942450 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260928 23:14:40.728645 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:40.728445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:51441)
0928 23:14:40.728557 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:40.728598 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260928 23:14:40.808212 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:40.808003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:51567)
0928 23:14:40.808130 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:40.808169 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260928 23:14:40.943073 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:40.942877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:53499)
0928 23:14:40.943002 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:40.943036 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260928 23:14:41.415881 15691 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:41.415985 15691 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:41.416157 15691 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
W20260928 23:14:41.416501 15290 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
W20260928 23:14:41.416579 15293 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:41.416635 15293 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb; no voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f
I20260928 23:14:41.416716 15691 raft_consensus.cc:2726] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:14:41.729283 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:41.729040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:33739)
0928 23:14:41.729168 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:41.729203 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 23:14:41.808846 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:41.808655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:56007)
0928 23:14:41.808766 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:41.808804 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260928 23:14:41.943636 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:41.943433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:35187)
0928 23:14:41.943557 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:41.943596 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 23:14:42.729959 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:42.729734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:35925)
0928 23:14:42.729851 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:42.729901 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
I20260928 23:14:42.809494 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:42.809277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:51591)
0928 23:14:42.809405 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:42.809450 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260928 23:14:42.944180 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:42.943983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:35937)
0928 23:14:42.944107 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:42.944143 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260928 23:14:43.730643 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:43.730419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:56301)
0928 23:14:43.730549 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:43.730596 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 23:14:43.810084 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:43.809879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:50047)
0928 23:14:43.810001 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:43.810038 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 23:14:43.944705 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:43.944528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:45213)
0928 23:14:43.944629 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:43.944656 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 23:14:44.317029 15701 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:44.317126 15701 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:44.317312 15701 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
W20260928 23:14:44.317679 15290 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
W20260928 23:14:44.317783 15293 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:44.317823 15293 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb; no voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f
I20260928 23:14:44.317888 15701 raft_consensus.cc:2726] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:14:44.731335 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:44.731061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:36679)
0928 23:14:44.731247 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:44.731290 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260928 23:14:44.810678 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:44.810458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:55279)
0928 23:14:44.810578 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:44.810619 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 23:14:44.945324 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:44.945068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:60305)
0928 23:14:44.945240 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:44.945283 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260928 23:14:45.731952 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:45.731731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:50779)
0928 23:14:45.731871 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:45.731906 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260928 23:14:45.811226 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:45.811037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:60481)
0928 23:14:45.811151 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:45.811186 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 23:14:45.945873 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:45.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:35583)
0928 23:14:45.945798 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:45.945834 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260928 23:14:46.732668 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:46.732425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:47823)
0928 23:14:46.732560 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:46.732602 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260928 23:14:46.811864 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:46.811641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:42029)
0928 23:14:46.811776 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:46.811818 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 23:14:46.946554 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:46.946287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:54237)
0928 23:14:46.946453 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:46.946506 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260928 23:14:47.649858 15711 raft_consensus.cc:482] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:47.649959 15711 raft_consensus.cc:504] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cda40f5e3f461aa82e6aa51766ec2f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43813 } } peers { permanent_uuid: "209b754129804c90bc7dd663fa6209cb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34825 } } peers { permanent_uuid: "c2dd04287b934fd7acf7c93f3226e16b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40499 } }
I20260928 23:14:47.650137 15711 leader_election.cc:288] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813), c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499)
W20260928 23:14:47.650509 15290 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d2cda40f5e3f461aa82e6aa51766ec2f (127.0.98.2:43813): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
W20260928 23:14:47.650592 15293 leader_election.cc:341] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2dd04287b934fd7acf7c93f3226e16b (127.0.98.3:40499): Not found: Tablet not found: 150d66abd54c433c8fd07158c64efda5
I20260928 23:14:47.650636 15293 leader_election.cc:302] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 209b754129804c90bc7dd663fa6209cb; no voters: c2dd04287b934fd7acf7c93f3226e16b, d2cda40f5e3f461aa82e6aa51766ec2f
I20260928 23:14:47.650717 15711 raft_consensus.cc:2726] T 150d66abd54c433c8fd07158c64efda5 P 209b754129804c90bc7dd663fa6209cb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 23:14:47.733489 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:47.733158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:38161)
0928 23:14:47.733386 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:47.733439 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260928 23:14:47.812500 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:47.812286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:40373)
0928 23:14:47.812418 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:47.812457 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 23:14:47.947201 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:47.946982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:60135)
0928 23:14:47.947114 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:47.947157 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260928 23:14:48.734220 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:48.733992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:39737)
0928 23:14:48.734131 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:48.734173 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 23:14:48.813410 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:48.813151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:38635)
0928 23:14:48.813316 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:48.813362 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260928 23:14:48.947934 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:48.947693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:48719)
0928 23:14:48.947845 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:48.947891 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260928 23:14:49.734915 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:49.734674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:35083)
0928 23:14:49.734821 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:49.734867 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 23:14:49.814069 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:49.813864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:59195)
0928 23:14:49.813987 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:49.814027 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260928 23:14:49.948609 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:49.948390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:53711)
0928 23:14:49.948516 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:49.948562 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260928 23:14:50.735639 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:50.735405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:41589)
0928 23:14:50.735548 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:50.735593 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 23:14:50.814795 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:50.814548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:47819)
0928 23:14:50.814697 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:50.814745 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260928 23:14:50.949365 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:50.949091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:38363)
0928 23:14:50.949264 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:50.949308 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260928 23:14:51.736328 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:51.736116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.1:60287)
0928 23:14:51.736253 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:51.736283 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260928 23:14:51.815496 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:51.815292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.2:50457)
0928 23:14:51.815420 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:51.815453 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260928 23:14:51.944779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15245
I20260928 23:14:51.949079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15409
I20260928 23:14:51.950100 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:51.949863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35227 (local address 127.0.98.3:42689)
0928 23:14:51.950022 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:51.950053 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":52,"threads_started":1}
I20260928 23:14:51.952903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15541
2026-09-28T23:14:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18075 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:14:52 dist-test-slave-6vgd krb5kdc[15729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:14:52 dist-test-slave-6vgd krb5kdc[15729](info): set up 2 sockets
Sep 28 23:14:52 dist-test-slave-6vgd krb5kdc[15729](info): commencing operation
krb5kdc: starting...
W20260928 23:14:52.905123 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:14:52 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:14:53.020854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:53.099963 15742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:53.100183 15742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:53.100212 15742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:53.101420 15742 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:14:53.101461 15742 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:14:53.101478 15742 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:14:53.101493 15742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:14:53.101508 15742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:53.101526 15742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:53.101544 15742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:14:53.101567 15742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:14:53.103104 15742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:53.103374 15742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:53.103588 15742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:14:53.108520 15742 server_base.cc:1034] running on GCE node
W20260928 23:14:53.108662 15750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.108598 15747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.108596 15748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:53.109030 15742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:53.109350 15742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:53.110486 15742 hybrid_clock.cc:648] HybridClock initialized: now 1790637293110476 us; error 30 us; skew 500 ppm
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:53.112913 15742 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:14:53.113389 15742 webserver.cc:460] Webserver started at http://127.0.98.62:35769/ using document root <none> and password file <none>
I20260928 23:14:53.113588 15742 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:53.113628 15742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:53.113721 15742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:53.114734 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "42aa20d7a2f84230a225ff4ea3c98f28"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "59bd9bd647d5d8c936a1793683c799ac"
server_key_iv: "68c38a65639e0cf24afae32f776be787"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.115018 15742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "42aa20d7a2f84230a225ff4ea3c98f28"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "59bd9bd647d5d8c936a1793683c799ac"
server_key_iv: "68c38a65639e0cf24afae32f776be787"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.116225 15742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:14:53.116833 15757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.116972 15742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:53.117053 15742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "42aa20d7a2f84230a225ff4ea3c98f28"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "59bd9bd647d5d8c936a1793683c799ac"
server_key_iv: "68c38a65639e0cf24afae32f776be787"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.117110 15742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:53.138228 15742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:53.139947 15742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:53.140056 15742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:53.143738 15742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35083
I20260928 23:14:53.143778 15809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35083 every 8 connection(s)
I20260928 23:14:53.144129 15742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:14:53.144984 15810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:53.145612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15742
I20260928 23:14:53.145726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:14:53.145905 392 external_mini_cluster.cc:1428] Setting key 7397b1fc6dfff2e31c8b531ca9edb386
I20260928 23:14:53.147447 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap starting.
I20260928 23:14:53.148121 15810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:53.148438 15810 log.cc:824] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:53.149058 15810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: No bootstrap required, opened a new log
I20260928 23:14:53.151218 15810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:14:53.151384 15810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:53.151415 15810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Initialized, Role: FOLLOWER
I20260928 23:14:53.151538 15810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:14:53.151608 15810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:14:53.151643 15810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:14:53.151682 15810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:53.152299 15810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:14:53.152405 15810 leader_election.cc:302] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42aa20d7a2f84230a225ff4ea3c98f28; no voters:
I20260928 23:14:53.152565 15810 leader_election.cc:288] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:14:53.152925 15810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:14:53.152604 15815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:53.153493 15815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 LEADER]: Becoming Leader. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Running, Role: LEADER
I20260928 23:14:53.153703 15815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:53.154194 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42aa20d7a2f84230a225ff4ea3c98f28. Latest consensus state: current_term: 1 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 23:14:53.154284 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:53.154397 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
W20260928 23:14:53.154727 15829 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:14:53.154776 15829 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:14:53.154825 15830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:14:53.154641 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:14:53.154986 15830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:14:53.156685 15830 catalog_manager.cc:1347] Generated new cluster ID: 1f3900be73864bc9929e61b6a5ed8ffa
I20260928 23:14:53.156725 15830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:14:53.158133 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.146402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.0.98.62:35083)
0928 23:14:53.147622 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.147631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.147663 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:14:53.148204 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.148214 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.148286 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.148455 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:53.149052 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.150458 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.151164 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.151748 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.151856 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.154620 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:53.154625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:53.154625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:53.154637 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:53.156560 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.156948 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.156951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.156952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.157046 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.157273 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.157274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.157275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.157548 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:53.157684 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.157812 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.157939 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":69,"threads_started":1}
I20260928 23:14:53.173475 15830 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:14:53.174017 15830 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:14:53.179105 15830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Generated new TSK 0
I20260928 23:14:53.179329 15830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:14:53.204252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:53.284269 15836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:53.284495 15836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:53.284530 15836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:53.285789 15836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:53.285837 15836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:53.285876 15836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:14:53.287391 15836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:53.287655 15836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:53.287881 15836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:14:53.292950 15836 server_base.cc:1034] running on GCE node
W20260928 23:14:53.292881 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.292881 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.292907 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:53.293349 15836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:53.293598 15836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:53.294759 15836 hybrid_clock.cc:648] HybridClock initialized: now 1790637293294745 us; error 28 us; skew 500 ppm
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:53.297266 15836 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:14:53.297703 15836 webserver.cc:460] Webserver started at http://127.0.98.1:41903/ using document root <none> and password file <none>
I20260928 23:14:53.297917 15836 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:53.297967 15836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:53.298095 15836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:53.299181 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "83ef3357bd4740c1b984c09149aa45e5"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "d64dbe0f6e90984ded1b9185e4aa1897"
server_key_iv: "8e5346dd71cf508be3bd80b7ad29c44b"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.299507 15836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83ef3357bd4740c1b984c09149aa45e5"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "d64dbe0f6e90984ded1b9185e4aa1897"
server_key_iv: "8e5346dd71cf508be3bd80b7ad29c44b"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.300741 15836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:14:53.301460 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.301610 15836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:53.301697 15836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "83ef3357bd4740c1b984c09149aa45e5"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "d64dbe0f6e90984ded1b9185e4aa1897"
server_key_iv: "8e5346dd71cf508be3bd80b7ad29c44b"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.301759 15836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:53.311805 15836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:53.313694 15836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:53.313841 15836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:53.314056 15836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:53.314383 15836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:53.314419 15836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.314446 15836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:53.314464 15836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.320159 15836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260928 23:14:53.320191 15964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260928 23:14:53.320508 15836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:53.329403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15836
I20260928 23:14:53.329517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:14:53.329636 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.321347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43277 (local address 127.0.98.62:35083)
0928 23:14:53.321418 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.321420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.321591 (+ 171us) server_negotiation.cc:407] Connection header received
0928 23:14:53.322149 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.322150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.322162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.322224 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:53.323186 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.323706 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.324711 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.324918 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.324979 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.327067 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:53.327069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:53.327069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:53.327076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:53.328236 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.328594 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.328595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.328595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.328640 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.328945 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.328946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.328946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.329076 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:53.329140 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.329447 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.329545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:14:53.329713 392 external_mini_cluster.cc:1428] Setting key fc67942544bab267c731bbafce8032bd
I20260928 23:14:53.329793 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.321319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:43277)
0928 23:14:53.321474 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.321513 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.322009 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.322283 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.322286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.322907 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:53.323105 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.323110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.323782 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.323782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.324618 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.324620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.324985 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.324986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.325092 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.325591 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:53.325599 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:53.326980 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:53.328310 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.328317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.328319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.328543 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.328718 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.328720 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.328720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.328859 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.329135 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:53.329138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:53.329313 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.329411 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.329579 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260928 23:14:53.330296 15965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:14:53.330411 15965 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:53.330586 15965 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
I20260928 23:14:53.331126 15774 ts_manager.cc:194] Registered new tserver with Master: 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473)
I20260928 23:14:53.331898 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43277
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:14:53.354525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:53.433297 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:53.433496 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:53.433518 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:53.434614 15970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:53.434652 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:53.434676 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:14:53.436055 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:53.436259 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:53.436442 15970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:14:53.441372 15970 server_base.cc:1034] running on GCE node
W20260928 23:14:53.441322 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.441350 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.441339 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:53.441743 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:53.441967 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:53.443097 15970 hybrid_clock.cc:648] HybridClock initialized: now 1790637293443086 us; error 30 us; skew 500 ppm
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:53.445428 15970 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:14:53.445843 15970 webserver.cc:460] Webserver started at http://127.0.98.2:34073/ using document root <none> and password file <none>
I20260928 23:14:53.446046 15970 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:53.446091 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:53.446208 15970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:53.447232 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "0322442a1d20266d23845c8704cf98d2"
server_key_iv: "28a89e1c336b051e9f134c31e1501c46"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.447554 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "0322442a1d20266d23845c8704cf98d2"
server_key_iv: "28a89e1c336b051e9f134c31e1501c46"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.448796 15970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:14:53.449502 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.449628 15970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:14:53.449704 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "0322442a1d20266d23845c8704cf98d2"
server_key_iv: "28a89e1c336b051e9f134c31e1501c46"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.449769 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:53.467706 15970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:53.469772 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:53.469907 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:53.470118 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:53.470405 15970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:53.470438 15970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.470458 15970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:53.470481 15970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.475898 15970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34095
I20260928 23:14:53.475962 16098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34095 every 8 connection(s)
I20260928 23:14:53.476240 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:14:53.479015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 15970
I20260928 23:14:53.479110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:14:53.479302 392 external_mini_cluster.cc:1428] Setting key 29086e00370a0c4709ae76ad2ee5b2f8
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:53.485011 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.476883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49363 (local address 127.0.98.62:35083)
0928 23:14:53.476932 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.476934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.477116 (+ 182us) server_negotiation.cc:407] Connection header received
0928 23:14:53.477675 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.477676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.477687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.477738 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:53.478644 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.479222 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.480353 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.480547 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.480601 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.482585 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:53.482586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:53.482587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:53.482594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:53.483809 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.484164 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.484165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.484165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.484218 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.484424 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.484425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.484425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.484536 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:53.484592 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.484838 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.484923 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:14:53.485271 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.476848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:49363)
0928 23:14:53.477009 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.477044 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.477579 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.477808 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.477812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.478359 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:53.478552 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.478557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.479308 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.479309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.480267 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.480270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.480614 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.480615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.480736 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.481198 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:53.481231 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:53.482503 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:53.483882 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.483889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.483892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.484102 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.484269 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.484270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.484270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.484372 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.484587 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:53.484592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:53.484725 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.484811 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.485020 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260928 23:14:53.485730 16099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:14:53.485817 16099 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:53.485993 16099 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 23:14:53.486343 15774 ts_manager.cc:194] Registered new tserver with Master: 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
I20260928 23:14:53.486697 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:14:53.502020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:14:53.579224 16104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:14:53.579422 16104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:14:53.579443 16104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:14:53.580495 16104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:14:53.580526 16104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:14:53.580550 16104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:14:53.581921 16104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:14:53.582095 16104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:14:53.582269 16104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:14:53.586802 16112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.586802 16109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:14:53.586952 16110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:14:53.587090 16104 server_base.cc:1034] running on GCE node
I20260928 23:14:53.587304 16104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:14:53.587527 16104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:14:53.588656 16104 hybrid_clock.cc:648] HybridClock initialized: now 1790637293588640 us; error 33 us; skew 500 ppm
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:14:53.590939 16104 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:14:53.591388 16104 webserver.cc:460] Webserver started at http://127.0.98.3:43427/ using document root <none> and password file <none>
I20260928 23:14:53.591584 16104 fs_manager.cc:359] Metadata directory not provided
I20260928 23:14:53.591630 16104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:14:53.591755 16104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:14:53.592726 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "2a697aff710d4da3848efab2b0e94a9e"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "f71d8623247e791e52fdf294fffb7dfe"
server_key_iv: "a4f5bc2c94ecb7b3e98d066025e8d420"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.593039 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2a697aff710d4da3848efab2b0e94a9e"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "f71d8623247e791e52fdf294fffb7dfe"
server_key_iv: "a4f5bc2c94ecb7b3e98d066025e8d420"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.594288 16104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:14:53.594937 16119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.595078 16104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:14:53.595155 16104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "2a697aff710d4da3848efab2b0e94a9e"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "f71d8623247e791e52fdf294fffb7dfe"
server_key_iv: "a4f5bc2c94ecb7b3e98d066025e8d420"
server_key_version: "encryptionkey@0"
I20260928 23:14:53.595225 16104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:14:53.603924 16104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:14:53.605815 16104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:14:53.605937 16104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:14:53.606110 16104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:14:53.606379 16104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:14:53.606412 16104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.606431 16104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:14:53.606453 16104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:14:53.611624 16104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36163
I20260928 23:14:53.611687 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36163 every 8 connection(s)
I20260928 23:14:53.611933 16104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:14:53.616290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16104
I20260928 23:14:53.616398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:14:53.616582 392 external_mini_cluster.cc:1428] Setting key dd37ac090e54533478d7d8bed5d157d4
Sep 28 23:14:53 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:14:53.619879 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.612556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47075 (local address 127.0.98.62:35083)
0928 23:14:53.612595 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.612596 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.612779 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:14:53.613256 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.613257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.613265 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.613314 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:53.614120 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.614626 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.615525 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.615698 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.615742 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.617762 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:53.617763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:53.617763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:53.617770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:53.618823 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.619143 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.619143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.619144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.619185 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.619376 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.619376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.619377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.619474 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:53.619528 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.619762 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.619817 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:14:53.620021 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.612526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:47075)
0928 23:14:53.612671 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.612711 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.613155 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.613368 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.613371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.613882 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:14:53.614056 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.614061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.614686 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.614687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.615463 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.615466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.615755 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.615755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.615843 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.616253 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:14:53.616263 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:14:53.617688 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:14:53.618887 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.618893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.618894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.619089 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.619239 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:14:53.619239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:14:53.619240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:14:53.619324 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:14:53.619528 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:14:53.619531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:14:53.619635 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.619731 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.619840 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260928 23:14:53.620463 16233 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:14:53.620538 16233 heartbeater.cc:464] Registering TS with master...
I20260928 23:14:53.620692 16233 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
I20260928 23:14:53.621006 15774 ts_manager.cc:194] Registered new tserver with Master: 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163)
I20260928 23:14:53.621395 15774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47075
I20260928 23:14:53.628048 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:14:53.634382 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.629281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.0.98.62:35083)
0928 23:14:53.629330 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.629332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.629439 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:14:53.629497 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.629498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.629508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.629556 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:14:53.630063 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.630532 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.631214 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.631397 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.631449 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.632442 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:14:53.632444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:14:53.632444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:14:53.632450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:14:53.633424 (+ 974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.633700 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.633700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.633701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.633762 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:14:53.633917 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:14:53.633918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:14:53.633918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:14:53.634040 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:14:53.634125 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.634226 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.634300 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:14:53.635569 15774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:14:53.637059 15774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:14:53.642836 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.639042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:33308)
0928 23:14:53.639439 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.639449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.639492 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.639740 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.639741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.639743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:53.639790 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.639791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640950 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.640951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.641723 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.641726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642549 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.642623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642636 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.642705 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.642767 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":60,"threads_started":1}
I20260928 23:14:53.642977 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.638996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.0.98.1:42473)
0928 23:14:53.639166 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.639171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.639437 (+ 266us) server_negotiation.cc:407] Connection header received
0928 23:14:53.639508 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.639510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.639524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.639590 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:53.639810 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640484 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.641957 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642635 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642694 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642742 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.642849 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.642898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 23:14:53.643011 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.638865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.0.1:38808)
0928 23:14:53.639016 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.639052 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.639113 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.639338 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.639341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.639342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:53.639399 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.639400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640656 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.640657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.642092 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.642094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642814 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.642869 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642872 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.642917 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.642951 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":32,"thread_start_us":53,"threads_started":1}
I20260928 23:14:53.643190 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.638871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:55272)
0928 23:14:53.639379 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.639388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.639442 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.639685 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.639686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.639688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:53.639727 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.639728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640572 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.640573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.641882 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.641884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642696 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.642761 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642765 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.643080 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.643125 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":45,"thread_start_us":50,"threads_started":1}
I20260928 23:14:53.643203 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.639232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.0.98.2:34095)
0928 23:14:53.639378 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.639382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.639548 (+ 166us) server_negotiation.cc:407] Connection header received
0928 23:14:53.639588 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.639590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.639604 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.639668 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:53.639857 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640852 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.641801 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642469 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642543 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642598 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.642798 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.642841 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260928 23:14:53.643263 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.638932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.0.98.3:36163)
0928 23:14:53.639198 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.639202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.639212 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:14:53.639244 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.639246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.639259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.639321 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:53.639477 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.640561 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642154 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.642677 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.642731 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.642782 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.643148 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.643192 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260928 23:14:53.643810 15899 tablet_service.cc:1467] Processing CreateTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 (DEFAULT_TABLE table=test-table [id=5fe2941043c846b39b2f325c1b8dfe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:53.643810 16033 tablet_service.cc:1467] Processing CreateTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 (DEFAULT_TABLE table=test-table [id=5fe2941043c846b39b2f325c1b8dfe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:53.643868 16167 tablet_service.cc:1467] Processing CreateTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 (DEFAULT_TABLE table=test-table [id=5fe2941043c846b39b2f325c1b8dfe48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:14:53.644136 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:53.644136 15899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:53.644140 16167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:14:53.646502 16247 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Bootstrap starting.
I20260928 23:14:53.647168 16247 tablet_bootstrap.cc:654] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:53.647181 16249 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap starting.
I20260928 23:14:53.647770 16247 log.cc:824] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:53.647801 16249 tablet_bootstrap.cc:654] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:53.648031 16248 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Bootstrap starting.
I20260928 23:14:53.648299 16249 log.cc:824] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:53.648486 16247 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: No bootstrap required, opened a new log
I20260928 23:14:53.648556 16247 ts_tablet_manager.cc:1397] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:53.648643 16248 tablet_bootstrap.cc:654] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Neither blocks nor log segments found. Creating new log.
I20260928 23:14:53.648913 16249 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: No bootstrap required, opened a new log
I20260928 23:14:53.648976 16249 ts_tablet_manager.cc:1397] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:14:53.649168 16248 log.cc:824] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Log is configured to *not* fsync() on all Append() calls
I20260928 23:14:53.649864 16248 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: No bootstrap required, opened a new log
I20260928 23:14:53.649964 16248 ts_tablet_manager.cc:1397] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:14:53.650231 16247 raft_consensus.cc:348] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.650364 16247 raft_consensus.cc:374] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:53.650396 16247 raft_consensus.cc:729] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a697aff710d4da3848efab2b0e94a9e, State: Initialized, Role: FOLLOWER
I20260928 23:14:53.650506 16247 consensus_queue.cc:260] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.650753 16247 ts_tablet_manager.cc:1428] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:14:53.650765 16233 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
I20260928 23:14:53.650767 16249 raft_consensus.cc:348] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.650916 16249 raft_consensus.cc:374] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:53.650947 16249 raft_consensus.cc:729] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Initialized, Role: FOLLOWER
I20260928 23:14:53.651062 16249 consensus_queue.cc:260] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.651316 16249 ts_tablet_manager.cc:1428] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:14:53.651336 15965 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
I20260928 23:14:53.651829 16248 raft_consensus.cc:348] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.651985 16248 raft_consensus.cc:374] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:14:53.652025 16248 raft_consensus.cc:729] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4d8a9149b54cfa9f1987e04c04ef02, State: Initialized, Role: FOLLOWER
I20260928 23:14:53.652118 16248 consensus_queue.cc:260] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.652381 16248 ts_tablet_manager.cc:1428] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:14:53.652433 16099 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
W20260928 23:14:53.726763 16100 tablet.cc:2369] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:53.820894 16254 raft_consensus.cc:482] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:14:53.821013 16254 raft_consensus.cc:504] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
W20260928 23:14:53.821235 15966 tablet.cc:2369] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:53.821470 16254 leader_election.cc:288] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163), 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
I20260928 23:14:53.825194 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.821733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43563 (local address 127.0.98.2:34095)
0928 23:14:53.821774 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.821776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.821891 (+ 115us) server_negotiation.cc:407] Connection header received
0928 23:14:53.821951 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.821952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.821963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.822003 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:53.822096 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.822772 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824223 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.824821 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.824911 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.825084 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.825122 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:14:53.825228 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.821622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.98.1:59047)
0928 23:14:53.821669 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.821680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.821722 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.821871 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.821871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.821873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:53.821945 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.821947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.822830 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.822831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.824111 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.824114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.824903 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.825012 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.825019 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.825070 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.825112 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":52}
I20260928 23:14:53.825240 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.821690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59047 (local address 127.0.98.3:36163)
0928 23:14:53.821745 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.821746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.821755 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:14:53.821782 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.821783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.821794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.821853 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:14:53.822006 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.822754 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824223 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.824821 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824902 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.824911 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.825084 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.825123 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:14:53.825241 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.821626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.1:43563)
0928 23:14:53.821830 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.821838 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:14:53.821896 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:14:53.822004 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:14:53.822004 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:14:53.822006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:14:53.822055 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.822056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.822825 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.822826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.824111 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:14:53.824114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.824905 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.824906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:14:53.825012 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.825020 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 23:14:53.825070 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:14:53.825112 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":159,"thread_start_us":78,"threads_started":1}
I20260928 23:14:53.825568 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648de6fa9d7b4ec5a4b43c71b14c8a15" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a697aff710d4da3848efab2b0e94a9e" is_pre_election: true
I20260928 23:14:53.825730 16187 raft_consensus.cc:2454] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 0.
I20260928 23:14:53.825850 16053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648de6fa9d7b4ec5a4b43c71b14c8a15" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" is_pre_election: true
I20260928 23:14:53.825978 16053 raft_consensus.cc:2454] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 0.
I20260928 23:14:53.825961 15852 leader_election.cc:302] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a697aff710d4da3848efab2b0e94a9e, 83ef3357bd4740c1b984c09149aa45e5; no voters:
I20260928 23:14:53.826087 16254 raft_consensus.cc:2781] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:14:53.826155 16254 raft_consensus.cc:482] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:14:53.826179 16254 raft_consensus.cc:3037] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:53.826845 16254 raft_consensus.cc:504] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.826962 16254 leader_election.cc:288] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 1 election: Requested vote from peers 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163), 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
I20260928 23:14:53.827099 16053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648de6fa9d7b4ec5a4b43c71b14c8a15" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
I20260928 23:14:53.827139 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "648de6fa9d7b4ec5a4b43c71b14c8a15" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a697aff710d4da3848efab2b0e94a9e"
I20260928 23:14:53.827178 16053 raft_consensus.cc:3037] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:53.827205 16187 raft_consensus.cc:3037] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:14:53.827877 16187 raft_consensus.cc:2454] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 1.
I20260928 23:14:53.827881 16053 raft_consensus.cc:2454] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 1.
I20260928 23:14:53.828045 15853 leader_election.cc:302] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4d8a9149b54cfa9f1987e04c04ef02, 83ef3357bd4740c1b984c09149aa45e5; no voters:
I20260928 23:14:53.828123 16254 raft_consensus.cc:2781] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:14:53.828281 16254 raft_consensus.cc:686] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 LEADER]: Becoming Leader. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Running, Role: LEADER
I20260928 23:14:53.828366 16254 consensus_queue.cc:237] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:14:53.829147 15773 catalog_manager.cc:5495] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83ef3357bd4740c1b984c09149aa45e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } }
W20260928 23:14:53.862453 16234 tablet.cc:2369] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:14:53.869896 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.867807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.0.98.1:42473)
0928 23:14:53.867865 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:14:53.867866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:14:53.867874 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:14:53.867913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:14:53.867914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:14:53.867933 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:14:53.867988 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:14:53.868100 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.868563 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.869283 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:14:53.869473 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:14:53.869518 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:14:53.869643 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:14:53.869738 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:14:53.869792 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:14:53.869804 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:14:53.869819 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:14:53.871549 16053 raft_consensus.cc:1261] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Refusing update from remote peer 83ef3357bd4740c1b984c09149aa45e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:53.871552 16187 raft_consensus.cc:1261] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Refusing update from remote peer 83ef3357bd4740c1b984c09149aa45e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:14:53.871915 16257 consensus_queue.cc:1035] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:53.872033 16254 consensus_queue.cc:1035] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:14:53.875450 16262 mvcc.cc:205] Tried to move back new op lower bound from 7334450355695525888 to 7334450355521552384. Current Snapshot: MvccSnapshot[applied={T|T < 7334450355695525888 or (T in {7334450355695525888})}]
I20260928 23:14:53.876546 15774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:53.876643 15774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:14:53.877360 15774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Sending DeleteTablet for 3 replicas of tablet 648de6fa9d7b4ec5a4b43c71b14c8a15
I20260928 23:14:53.877656 16167 tablet_service.cc:1514] Processing DeleteTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38808
I20260928 23:14:53.877666 16033 tablet_service.cc:1514] Processing DeleteTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33308
I20260928 23:14:53.877868 15899 tablet_service.cc:1514] Processing DeleteTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55272
I20260928 23:14:53.877904 16270 tablet_replica.cc:330] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: stopping tablet replica
I20260928 23:14:53.878077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15836
I20260928 23:14:53.878096 16271 tablet_replica.cc:330] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: stopping tablet replica
I20260928 23:14:53.878209 16271 raft_consensus.cc:2229] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:14:53.878299 16271 raft_consensus.cc:2258] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:53.879664 16270 raft_consensus.cc:2229] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:53.879786 16270 raft_consensus.cc:2258] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:53.880164 16270 ts_tablet_manager.cc:1905] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:53.881304 16269 tablet_replica.cc:330] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: stopping tablet replica
I20260928 23:14:53.881402 16269 raft_consensus.cc:2229] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:14:53.881501 16269 raft_consensus.cc:2258] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:14:53.881881 16269 ts_tablet_manager.cc:1905] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:14:53.881870 16270 ts_tablet_manager.cc:1918] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:53.881940 16270 log.cc:1196] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/648de6fa9d7b4ec5a4b43c71b14c8a15
I20260928 23:14:53.882184 16270 ts_tablet_manager.cc:1939] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 2a697aff710d4da3848efab2b0e94a9e: Deleting consensus metadata
I20260928 23:14:53.882552 15758 catalog_manager.cc:4841] TS 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163): tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 (table test-table [id=5fe2941043c846b39b2f325c1b8dfe48]) successfully deleted
I20260928 23:14:53.883190 16269 ts_tablet_manager.cc:1918] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:14:53.883253 16269 log.cc:1196] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/648de6fa9d7b4ec5a4b43c71b14c8a15
I20260928 23:14:53.883487 16269 ts_tablet_manager.cc:1939] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting consensus metadata
W20260928 23:14:53.883517 15759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42473 (error 108)
I20260928 23:14:53.883600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15970
I20260928 23:14:53.883934 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.883830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:55280)
0928 23:14:53.883868 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.883905 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:14:53.884028 15759 catalog_manager.cc:4584] TS 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
I20260928 23:14:53.887625 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:14:53.887524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:33318)
0928 23:14:53.887566 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:14:53.887596 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:14:53.887624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16104
I20260928 23:14:53.891836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 15742
I20260928 23:15:13.896481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--webserver_interface=127.0.98.62
--webserver_port=35769
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:13.974815 16273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:13.975028 16273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:13.975056 16273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:13.976171 16273 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:15:13.976208 16273 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:15:13.976225 16273 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:15:13.976240 16273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:13.976257 16273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:13.976281 16273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:13.976298 16273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:13.976321 16273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:13.977813 16273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:13.978051 16273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:13.978247 16273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:13.986122 16282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:13.986200 16273 server_base.cc:1034] running on GCE node
W20260928 23:15:13.986202 16279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:13.986122 16280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:13.986598 16273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:13.986909 16273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:13.988046 16273 hybrid_clock.cc:648] HybridClock initialized: now 1790637313988038 us; error 28 us; skew 500 ppm
Sep 28 23:15:13 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:13.995929 16273 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:15:13.996342 16273 webserver.cc:460] Webserver started at http://127.0.98.62:35769/ using document root <none> and password file <none>
I20260928 23:15:13.996534 16273 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:13.996582 16273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:13.997933 16273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:15:13.998495 16289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:13.998636 16273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:13.998718 16273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "42aa20d7a2f84230a225ff4ea3c98f28"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "59bd9bd647d5d8c936a1793683c799ac"
server_key_iv: "68c38a65639e0cf24afae32f776be787"
server_key_version: "encryptionkey@0"
I20260928 23:15:13.999100 16273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:14.013423 16273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:14.015061 16273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:14.015187 16273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:14.018807 16273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35083
I20260928 23:15:14.018838 16341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35083 every 8 connection(s)
I20260928 23:15:14.019176 16273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:14.020533 16342 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:15:14.021198 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap starting.
I20260928 23:15:14.021234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16273
I20260928 23:15:14.021507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=41903
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:15:14.029479 16342 log.cc:824] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:14.031168 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:14.031409 16342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap complete.
I20260928 23:15:14.033473 16342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:14.033689 16342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Initialized, Role: FOLLOWER
I20260928 23:15:14.033833 16342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:14.033890 16342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:14.033932 16342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:14.033977 16342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:15:14.034950 16342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:14.035035 16342 leader_election.cc:302] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42aa20d7a2f84230a225ff4ea3c98f28; no voters:
I20260928 23:15:14.035192 16342 leader_election.cc:288] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 23:15:14.035235 16346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:15:14.035343 16346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 LEADER]: Becoming Leader. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Running, Role: LEADER
I20260928 23:15:14.035445 16342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:14.035430 16346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:14.035702 16347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 23:15:14.035715 16348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42aa20d7a2f84230a225ff4ea3c98f28. Latest consensus state: current_term: 2 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 23:15:14.035799 16347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:14.035828 16348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:14.036378 16352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:14.037120 16352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:14.037272 16352 catalog_manager.cc:1259] Loaded cluster ID: 1f3900be73864bc9929e61b6a5ed8ffa
I20260928 23:15:14.037307 16352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:14.038479 16352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:14.038655 16352 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Loaded TSK: 0
I20260928 23:15:14.050777 16352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Generated new TSK 1
I20260928 23:15:14.050863 16352 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:15:14.104305 16344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:14.104506 16344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:14.104528 16344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:14.105734 16344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:14.105777 16344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:14.105803 16344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:14.107242 16344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41903
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:14.107460 16344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:14.107673 16344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:14.113373 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:14.113384 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:14.113505 16344 server_base.cc:1034] running on GCE node
W20260928 23:15:14.113384 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:14.113804 16344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:14.114022 16344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:14.115157 16344 hybrid_clock.cc:648] HybridClock initialized: now 1790637314115144 us; error 30 us; skew 500 ppm
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:14.117619 16344 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:15:14.118162 16344 webserver.cc:460] Webserver started at http://127.0.98.1:41903/ using document root <none> and password file <none>
I20260928 23:15:14.118413 16344 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:14.118491 16344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:14.120157 16344 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:14.120832 16379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.120998 16344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:14.121079 16344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "83ef3357bd4740c1b984c09149aa45e5"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "d64dbe0f6e90984ded1b9185e4aa1897"
server_key_iv: "8e5346dd71cf508be3bd80b7ad29c44b"
server_key_version: "encryptionkey@0"
I20260928 23:15:14.121447 16344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:14.144595 16344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:14.146636 16344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:14.146754 16344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:14.146988 16344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:14.147403 16386 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:15:14.149243 16344 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:15:14.149287 16344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:15:14.149319 16344 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:15:14.149901 16344 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:15:14.149966 16344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:15:14.150031 16386 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap starting.
I20260928 23:15:14.156786 16344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260928 23:15:14.157143 16344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:14.160199 16386 log.cc:824] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:14.160490 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260928 23:15:14.162518 16386 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:14.162755 16386 tablet_bootstrap.cc:492] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap complete.
I20260928 23:15:14.163475 16386 ts_tablet_manager.cc:1397] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260928 23:15:14.164412 16386 raft_consensus.cc:348] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:15:14.164625 16386 raft_consensus.cc:729] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Initialized, Role: FOLLOWER
I20260928 23:15:14.164767 16386 consensus_queue.cc:260] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } }
I20260928 23:15:14.165026 16386 ts_tablet_manager.cc:1428] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:14.166456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16344
I20260928 23:15:14.166687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=34073
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:15:14.169727 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.157930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37035 (local address 127.0.98.62:35083)
0928 23:15:14.161396 (+ 3466us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.161403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.161428 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:15:14.161549 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.161554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.161593 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.161701 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:14.162324 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.163234 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.164232 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.164602 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.164692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.166688 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:14.166692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:14.166692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:14.166701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:14.168152 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.168510 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.168513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.168514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.168599 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.168846 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.168847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.168848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.169079 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:14.169152 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.169486 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.169594 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":67,"threads_started":1}
I20260928 23:15:14.170055 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.157859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:37035)
0928 23:15:14.160731 (+ 2872us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.160764 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.161197 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.161744 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.161748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.162028 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:14.162214 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.162218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.163333 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.163334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.164152 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.164155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.164678 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.164679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.164802 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.165315 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:14.165324 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:14.166591 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:14.168229 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.168236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.168238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.168439 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.168657 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.168658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.168658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.168789 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.169149 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:14.169153 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:14.169332 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.169433 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.169589 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2833,"thread_start_us":103,"threads_started":1}
I20260928 23:15:14.170704 16494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:15:14.170787 16494 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:14.170984 16494 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
I20260928 23:15:14.171623 16306 ts_manager.cc:194] Registered new tserver with Master: 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473)
I20260928 23:15:14.172602 16306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37035
I20260928 23:15:14.173486 16494 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
I20260928 23:15:14.176224 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.172203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:46852)
0928 23:15:14.173313 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.173354 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.173441 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.173680 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.173683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.173685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:14.173743 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.173745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.174579 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.174580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.175364 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.175366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.175952 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.175953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.176032 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.176045 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.176105 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.176158 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":80,"threads_started":1}
I20260928 23:15:14.176222 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.172240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.98.1:42473)
0928 23:15:14.172414 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.172418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.173434 (+ 1016us) server_negotiation.cc:407] Connection header received
0928 23:15:14.173522 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.173525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.173539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.173618 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:14.173809 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.174508 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.175441 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.175881 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.175941 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.175993 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.176123 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.176158 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260928 23:15:14.176563 16428 tablet_service.cc:1514] Processing DeleteTablet for tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:14:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46852
I20260928 23:15:14.177030 16503 tablet_replica.cc:330] stopping tablet replica
I20260928 23:15:14.177119 16503 raft_consensus.cc:2229] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:14.177182 16503 raft_consensus.cc:2258] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:14.177467 16503 ts_tablet_manager.cc:1905] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:14.179641 16503 ts_tablet_manager.cc:1918] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:14.179697 16503 log.cc:1196] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal/wals/648de6fa9d7b4ec5a4b43c71b14c8a15
I20260928 23:15:14.179873 16503 ts_tablet_manager.cc:1939] T 648de6fa9d7b4ec5a4b43c71b14c8a15 P 83ef3357bd4740c1b984c09149aa45e5: Deleting consensus metadata
I20260928 23:15:14.180305 16291 catalog_manager.cc:4841] TS 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473): tablet 648de6fa9d7b4ec5a4b43c71b14c8a15 (table test-table [id=5fe2941043c846b39b2f325c1b8dfe48]) successfully deleted
W20260928 23:15:14.245792 16500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:14.245990 16500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:14.246019 16500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:14.247139 16500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:14.247179 16500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:14.247215 16500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:14.248621 16500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34073
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:14.248842 16500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:14.249045 16500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:14.253799 16508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:14.253789 16509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:14.253808 16511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:14.253991 16500 server_base.cc:1034] running on GCE node
I20260928 23:15:14.254206 16500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:14.254448 16500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:14.255597 16500 hybrid_clock.cc:648] HybridClock initialized: now 1790637314255578 us; error 34 us; skew 500 ppm
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:14.257829 16500 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:15:14.258260 16500 webserver.cc:460] Webserver started at http://127.0.98.2:34073/ using document root <none> and password file <none>
I20260928 23:15:14.258466 16500 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:14.258526 16500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:14.259748 16500 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:14.260306 16518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.260463 16500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:14.260540 16500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "0322442a1d20266d23845c8704cf98d2"
server_key_iv: "28a89e1c336b051e9f134c31e1501c46"
server_key_version: "encryptionkey@0"
I20260928 23:15:14.260862 16500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:14.282672 16500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:14.284390 16500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:14.284528 16500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:14.284749 16500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:14.285036 16500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:14.285069 16500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.285099 16500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:14.285116 16500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.290469 16500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34095
I20260928 23:15:14.290529 16631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34095 every 8 connection(s)
I20260928 23:15:14.290838 16500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:14.291407 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16500
I20260928 23:15:14.291674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36163
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=43427
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:14.300688 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.291516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46517 (local address 127.0.98.62:35083)
0928 23:15:14.291573 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.291576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.291770 (+ 194us) server_negotiation.cc:407] Connection header received
0928 23:15:14.292302 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.292304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.292319 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.292371 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:14.293578 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.294079 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.295478 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.295671 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.295722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.298035 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:14.298037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:14.298038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:14.298046 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:14.299435 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.299793 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.299794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.299794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.299855 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.300081 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.300082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.300082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.300219 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:14.300290 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.300503 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.300594 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:15:14.300747 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.291495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:46517)
0928 23:15:14.291642 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.291684 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.292199 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.292449 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.292453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.293151 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:14.293466 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.293472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.294166 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.294167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.295400 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.295403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.295748 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.295750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.295905 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.296392 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:14.296402 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:14.297932 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:14.299511 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.299517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.299519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.299726 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.299932 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.299932 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.299933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.300027 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.300277 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:14.300280 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:14.300389 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.300465 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.300583 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260928 23:15:14.301194 16632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:15:14.301330 16632 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:14.301532 16632 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
I20260928 23:15:14.301959 16306 ts_manager.cc:194] Registered new tserver with Master: 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
I20260928 23:15:14.302345 16306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46517
W20260928 23:15:14.372303 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:14.372511 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:14.372534 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:14.373692 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:14.373732 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:14.373757 16635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:14.375164 16635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43427
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:14.375379 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:14.375603 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:14.380254 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:14.380363 16635 server_base.cc:1034] running on GCE node
W20260928 23:15:14.380244 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:14.380263 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:14.380605 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:14.380841 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:14.381976 16635 hybrid_clock.cc:648] HybridClock initialized: now 1790637314381965 us; error 28 us; skew 500 ppm
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:14.384361 16635 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:15:14.384770 16635 webserver.cc:460] Webserver started at http://127.0.98.3:43427/ using document root <none> and password file <none>
I20260928 23:15:14.384955 16635 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:14.385010 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:14.386307 16635 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:14.386840 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.386996 16635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:14.387075 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "2a697aff710d4da3848efab2b0e94a9e"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "f71d8623247e791e52fdf294fffb7dfe"
server_key_iv: "a4f5bc2c94ecb7b3e98d066025e8d420"
server_key_version: "encryptionkey@0"
I20260928 23:15:14.387385 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:14.398602 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:14.400321 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:14.400425 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:14.400589 16635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:14.400830 16635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:14.400858 16635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.400878 16635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:14.400889 16635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.406188 16635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36163
I20260928 23:15:14.406265 16763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36163 every 8 connection(s)
I20260928 23:15:14.406559 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 23:15:14 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:14.414948 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.407214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59973 (local address 127.0.98.62:35083)
0928 23:15:14.407305 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.407307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.407460 (+ 153us) server_negotiation.cc:407] Connection header received
0928 23:15:14.407916 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.407917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.407926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.407973 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:14.408845 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.409370 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.410289 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.410472 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.410513 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.412509 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:14.412510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:14.412511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:14.412516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:14.413715 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.414093 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.414094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.414094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.414146 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.414356 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.414357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.414357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.414473 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:14.414528 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.414782 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.414860 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:15:14.415124 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.407181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:59973)
0928 23:15:14.407346 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.407381 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.407841 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.408031 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.408033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.408580 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:14.408772 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.408777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.409441 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.409442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.410224 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.410226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.410523 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.410524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.410623 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.411108 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:14.411117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:14.412420 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:14.413794 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.413801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.413804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.414026 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.414201 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:14.414202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:14.414202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:14.414300 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:14.414529 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:14.414536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:14.414665 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.414753 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.414905 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 23:15:14.415525 16764 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35083
I20260928 23:15:14.415612 16764 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:14.415777 16764 heartbeater.cc:511] Master 127.0.98.62:35083 requested a full tablet report, sending...
I20260928 23:15:14.416113 16306 ts_manager.cc:194] Registered new tserver with Master: 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163)
I20260928 23:15:14.416460 16306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59973
I20260928 23:15:14.416533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16635
I20260928 23:15:14.421940 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.416914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.98.62:35083)
0928 23:15:14.416962 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.416963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.417127 (+ 164us) server_negotiation.cc:407] Connection header received
0928 23:15:14.417185 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.417186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.417193 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.417259 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:14.417775 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.418220 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.418884 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.419047 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.419086 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.420034 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:14.420035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:14.420035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:14.420041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:14.421161 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.421438 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.421439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.421439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.421485 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.421598 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.421599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.421599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.421678 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:14.421778 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.421808 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.421860 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 23:15:14.422603 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:14.425105 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.422878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.98.62:35083)
0928 23:15:14.422910 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.422912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.423008 (+ 96us) server_negotiation.cc:407] Connection header received
0928 23:15:14.423059 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.423060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.423086 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.423139 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:14.423200 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.423622 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.424437 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.424615 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.424663 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.424782 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:14.424824 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 23:15:14.425063 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38248: token expired
Metrics: {"server-negotiator.queue_time_us":14}
W20260928 23:15:14.425139 16497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38248: token expired
W20260928 23:15:14.425810 16239 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:35083)
I20260928 23:15:14.431833 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.426074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.0.98.62:35083)
0928 23:15:14.426120 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.426122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.426130 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:14.426206 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.426207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.426216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.426274 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:14.426802 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.427435 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.428113 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.428309 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.428359 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.429450 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:14.429451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:14.429452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:14.429458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:14.430741 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.431077 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.431078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.431079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.431150 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:14.431333 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:14.431334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:14.431335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:14.431476 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:14.431559 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.431660 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.431738 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:15:14.433377 16306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:14.434958 16306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:14.437114 16428 tablet_service.cc:1467] Processing CreateTablet for tablet ba3ac82f69434465b73ddfd6b152b11a (DEFAULT_TABLE table=test-table [id=26f843d8eed0486d8fd26517549a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:14.437337 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3ac82f69434465b73ddfd6b152b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:14.438825 16386 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap starting.
I20260928 23:15:14.439136 16386 tablet_bootstrap.cc:654] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:14.439672 16386 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: No bootstrap required, opened a new log
I20260928 23:15:14.439723 16386 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 23:15:14.439926 16386 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.440013 16386 raft_consensus.cc:374] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:14.440058 16386 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Initialized, Role: FOLLOWER
I20260928 23:15:14.440109 16386 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.440191 16386 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:14.440315 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.436636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:33032)
0928 23:15:14.436773 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.436783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.436833 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.437274 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.437275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.437277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:14.437315 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.437316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.438265 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.438266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.439122 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.439126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.440108 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.440109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.440174 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.440178 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.440206 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.440245 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 23:15:14.440466 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.436585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.0.1:36742)
0928 23:15:14.436634 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.436644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.436682 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.437011 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.437011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.437013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:14.437051 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.437053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.438533 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.438534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.439448 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.439450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.440283 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.440283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.440347 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.440350 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.440378 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.440413 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:15:14.440696 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.436918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.0.98.2:34095)
0928 23:15:14.437092 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.437096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.437106 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:14.437143 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.437145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.437160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.437277 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:14.437365 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.438193 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.439299 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.440014 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.440472 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.440548 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.440575 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.440617 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260928 23:15:14.440754 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.436670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.0.98.3:36163)
0928 23:15:14.436858 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.436862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.436873 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:15:14.436912 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.436915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.436929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.437006 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:14.437113 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.438424 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.439541 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.440207 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.440558 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.440614 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.440641 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.440681 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260928 23:15:14.441387 16566 tablet_service.cc:1467] Processing CreateTablet for tablet ba3ac82f69434465b73ddfd6b152b11a (DEFAULT_TABLE table=test-table [id=26f843d8eed0486d8fd26517549a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:14.441493 16698 tablet_service.cc:1467] Processing CreateTablet for tablet ba3ac82f69434465b73ddfd6b152b11a (DEFAULT_TABLE table=test-table [id=26f843d8eed0486d8fd26517549a15ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:14.441656 16566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3ac82f69434465b73ddfd6b152b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:14.441794 16698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3ac82f69434465b73ddfd6b152b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:14.444250 16773 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Bootstrap starting.
I20260928 23:15:14.444554 16774 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Bootstrap starting.
I20260928 23:15:14.444904 16773 tablet_bootstrap.cc:654] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:14.445168 16774 tablet_bootstrap.cc:654] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:14.445474 16773 log.cc:824] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:14.445758 16774 log.cc:824] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:14.446151 16773 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: No bootstrap required, opened a new log
I20260928 23:15:14.446208 16773 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:14.446378 16774 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: No bootstrap required, opened a new log
I20260928 23:15:14.446432 16774 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:15:14.447701 16773 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.447846 16773 raft_consensus.cc:374] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:14.447868 16773 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4d8a9149b54cfa9f1987e04c04ef02, State: Initialized, Role: FOLLOWER
I20260928 23:15:14.447880 16774 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.447966 16773 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.448027 16774 raft_consensus.cc:374] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:14.448060 16774 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a697aff710d4da3848efab2b0e94a9e, State: Initialized, Role: FOLLOWER
I20260928 23:15:14.448202 16773 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:14.448180 16774 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.448244 16632 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
I20260928 23:15:14.448443 16774 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:15:14.448467 16764 heartbeater.cc:503] Master 127.0.98.62:35083 was elected leader, sending a full tablet report...
W20260928 23:15:14.541373 16633 tablet.cc:2369] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:15:14.657109 16765 tablet.cc:2369] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:15:14.660996 16495 tablet.cc:2369] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:14.687269 16778 raft_consensus.cc:482] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:14.687423 16778 raft_consensus.cc:504] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.687790 16778 leader_election.cc:288] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095), 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473)
I20260928 23:15:14.690732 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.688051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52963 (local address 127.0.98.2:34095)
0928 23:15:14.688109 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.688111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.688122 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:15:14.688153 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.688154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.688165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.688439 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:14.688480 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.688991 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.689989 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.690420 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.690483 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.690494 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.690632 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.690666 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 23:15:14.690747 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.688193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51477 (local address 127.0.98.1:42473)
0928 23:15:14.688307 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.688308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.688316 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:14.688377 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.688377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.688386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.688438 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:14.688560 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.689031 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.689993 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.690440 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.690499 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.690510 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.690636 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.690678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260928 23:15:14.690748 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.687977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.3:52963)
0928 23:15:14.688030 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.688042 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.688088 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.688298 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.688299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.688301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:14.688378 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.688380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.689065 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.689066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.689897 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.689899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.690488 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.690489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.690570 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.690573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.690618 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.690658 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":35}
I20260928 23:15:14.690759 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.688107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.98.3:51477)
0928 23:15:14.688260 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.688270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:14.688313 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:14.688451 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:14.688451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:14.688453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:14.688504 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.688506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.689108 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.689109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.689908 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:14.689910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.690507 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.690508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:14.690579 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.690582 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:14.690627 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:15:14.690666 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 23:15:14.691115 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "2a697aff710d4da3848efab2b0e94a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" is_pre_election: true
I20260928 23:15:14.691273 16586 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a697aff710d4da3848efab2b0e94a9e in term 0.
I20260928 23:15:14.691504 16652 leader_election.cc:302] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a697aff710d4da3848efab2b0e94a9e, 3f4d8a9149b54cfa9f1987e04c04ef02; no voters:
I20260928 23:15:14.691617 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "2a697aff710d4da3848efab2b0e94a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ef3357bd4740c1b984c09149aa45e5" is_pre_election: true
I20260928 23:15:14.691673 16778 raft_consensus.cc:2781] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:14.691758 16778 raft_consensus.cc:482] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:14.691767 16448 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a697aff710d4da3848efab2b0e94a9e in term 0.
I20260928 23:15:14.691789 16778 raft_consensus.cc:3037] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:14.692489 16778 raft_consensus.cc:504] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.692610 16778 leader_election.cc:288] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [CANDIDATE]: Term 1 election: Requested vote from peers 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095), 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473)
I20260928 23:15:14.692826 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "2a697aff710d4da3848efab2b0e94a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83ef3357bd4740c1b984c09149aa45e5"
I20260928 23:15:14.692813 16586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "2a697aff710d4da3848efab2b0e94a9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
I20260928 23:15:14.692904 16586 raft_consensus.cc:3037] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:14.692903 16448 raft_consensus.cc:3037] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:14.693764 16448 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a697aff710d4da3848efab2b0e94a9e in term 1.
I20260928 23:15:14.693764 16586 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a697aff710d4da3848efab2b0e94a9e in term 1.
I20260928 23:15:14.693933 16652 leader_election.cc:302] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a697aff710d4da3848efab2b0e94a9e, 3f4d8a9149b54cfa9f1987e04c04ef02; no voters:
I20260928 23:15:14.694024 16778 raft_consensus.cc:2781] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:14.694154 16778 raft_consensus.cc:686] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 LEADER]: Becoming Leader. State: Replica: 2a697aff710d4da3848efab2b0e94a9e, State: Running, Role: LEADER
I20260928 23:15:14.694228 16778 consensus_queue.cc:237] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:14.695036 16306 catalog_manager.cc:5495] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2a697aff710d4da3848efab2b0e94a9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } health_report { overall_health: UNKNOWN } } }
I20260928 23:15:14.724174 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.722166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.0.98.3:36163)
0928 23:15:14.722214 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:14.722216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:14.722224 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:14.722264 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:14.722265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:14.722297 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:14.722361 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:14.722454 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.722909 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.723584 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:14.723783 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:14.723833 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:14.723926 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:14.724020 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:14.724070 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:14.724082 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:15:14.724098 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:15:14.725962 16586 raft_consensus.cc:1261] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Refusing update from remote peer 2a697aff710d4da3848efab2b0e94a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:14.725950 16448 raft_consensus.cc:1261] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Refusing update from remote peer 2a697aff710d4da3848efab2b0e94a9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:14.726380 16781 consensus_queue.cc:1035] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:14.726509 16778 consensus_queue.cc:1035] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:14.728673 16784 mvcc.cc:205] Tried to move back new op lower bound from 7334450441114775552 to 7334450440987996160. Current Snapshot: MvccSnapshot[applied={T|T < 7334450441114775552}]
I20260928 23:15:14.728816 16785 mvcc.cc:205] Tried to move back new op lower bound from 7334450441114775552 to 7334450440987996160. Current Snapshot: MvccSnapshot[applied={T|T < 7334450441114775552}]
I20260928 23:15:14.728869 16786 mvcc.cc:205] Tried to move back new op lower bound from 7334450441114775552 to 7334450440987996160. Current Snapshot: MvccSnapshot[applied={T|T < 7334450441114775552}]
I20260928 23:15:14.730921 16306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:14.731017 16306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:14.731681 16306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Sending DeleteTablet for 3 replicas of tablet ba3ac82f69434465b73ddfd6b152b11a
I20260928 23:15:14.732028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16344
I20260928 23:15:14.732033 16428 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3ac82f69434465b73ddfd6b152b11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46852
I20260928 23:15:14.732064 16566 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3ac82f69434465b73ddfd6b152b11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33032
I20260928 23:15:14.732298 16698 tablet_service.cc:1514] Processing DeleteTablet for tablet ba3ac82f69434465b73ddfd6b152b11a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36742
I20260928 23:15:14.732496 16795 tablet_replica.cc:330] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: stopping tablet replica
I20260928 23:15:14.732589 16795 raft_consensus.cc:2229] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:14.732682 16795 raft_consensus.cc:2258] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:14.733129 16795 ts_tablet_manager.cc:1905] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:14.734936 16795 ts_tablet_manager.cc:1918] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:14.735016 16795 log.cc:1196] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/ba3ac82f69434465b73ddfd6b152b11a
I20260928 23:15:14.735292 16795 ts_tablet_manager.cc:1939] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Deleting consensus metadata
I20260928 23:15:14.735728 16291 catalog_manager.cc:4841] TS 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095): tablet ba3ac82f69434465b73ddfd6b152b11a (table test-table [id=26f843d8eed0486d8fd26517549a15ff]) successfully deleted
W20260928 23:15:14.737071 16291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42473 (error 108)
I20260928 23:15:14.737466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16500
I20260928 23:15:14.737541 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.737426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.0.1:46856)
0928 23:15:14.737469 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.737508 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:15:14.742283 16291 catalog_manager.cc:4584] TS 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ba3ac82f69434465b73ddfd6b152b11a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42473: connect: Connection refused (error 111)
I20260928 23:15:14.742812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16635
I20260928 23:15:14.742806 16796 tablet_replica.cc:330] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: stopping tablet replica
I20260928 23:15:14.747473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16273
I20260928 23:15:14.747632 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:14.747500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.0.1:36768)
0928 23:15:14.747562 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:14.747594 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:36163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:15:34.752063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--webserver_interface=127.0.98.62
--webserver_port=35769
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:34.833410 16798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:34.833606 16798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:34.833635 16798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:34.834738 16798 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 23:15:34.834775 16798 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 23:15:34.834798 16798 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 23:15:34.834813 16798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:34.834831 16798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:34.834854 16798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:34.834872 16798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:34.834888 16798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:34.836330 16798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:34.836560 16798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:34.836750 16798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:15:34.841392 16798 server_base.cc:1034] running on GCE node
W20260928 23:15:34.841306 16806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:34.841306 16803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:34.841298 16804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:34.841732 16798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:34.841989 16798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:34.843125 16798 hybrid_clock.cc:648] HybridClock initialized: now 1790637334843112 us; error 26 us; skew 500 ppm
Sep 28 23:15:34 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:34.850180 16798 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:15:34.850596 16798 webserver.cc:460] Webserver started at http://127.0.98.62:35769/ using document root <none> and password file <none>
I20260928 23:15:34.850795 16798 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:34.850843 16798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:34.852077 16798 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:34.852633 16813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:34.852777 16798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:34.852854 16798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "42aa20d7a2f84230a225ff4ea3c98f28"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "59bd9bd647d5d8c936a1793683c799ac"
server_key_iv: "68c38a65639e0cf24afae32f776be787"
server_key_version: "encryptionkey@0"
I20260928 23:15:34.853175 16798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:34.877085 16798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:34.879148 16798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:34.879280 16798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:34.882944 16798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35083
I20260928 23:15:34.882999 16865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35083 every 8 connection(s)
I20260928 23:15:34.883317 16798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:34.884670 16866 sys_catalog.cc:263] Verifying existing consensus state
I20260928 23:15:34.885318 16866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap starting.
I20260928 23:15:34.887388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16798
I20260928 23:15:34.887699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42473
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=41903
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:15:34.894434 16866 log.cc:824] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:34.896590 16866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:34.896860 16866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Bootstrap complete.
I20260928 23:15:34.898911 16866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:34.899132 16866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Initialized, Role: FOLLOWER
I20260928 23:15:34.899282 16866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:34.899358 16866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:34.899406 16866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:34.899451 16866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 2 FOLLOWER]: Advancing to term 3
I20260928 23:15:34.900388 16866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:34.900486 16866 leader_election.cc:302] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42aa20d7a2f84230a225ff4ea3c98f28; no voters:
I20260928 23:15:34.900638 16866 leader_election.cc:288] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 23:15:34.900699 16870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 23:15:34.900879 16866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:34.900900 16870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [term 3 LEADER]: Becoming Leader. State: Replica: 42aa20d7a2f84230a225ff4ea3c98f28, State: Running, Role: LEADER
I20260928 23:15:34.900982 16870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } }
I20260928 23:15:34.901263 16871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 23:15:34.901278 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42aa20d7a2f84230a225ff4ea3c98f28. Latest consensus state: current_term: 3 leader_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42aa20d7a2f84230a225ff4ea3c98f28" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35083 } } }
I20260928 23:15:34.901352 16871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:34.901360 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:34.901986 16878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:34.902734 16878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:34.902824 16878 catalog_manager.cc:1259] Loaded cluster ID: 1f3900be73864bc9929e61b6a5ed8ffa
I20260928 23:15:34.902860 16878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:34.903654 16878 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:34.903808 16878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Loaded TSK: 0
I20260928 23:15:34.903844 16878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42aa20d7a2f84230a225ff4ea3c98f28: Loaded TSK: 1
F20260928 23:15:34.904039 16878 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f59d1eb89ad google::LogMessage::Fail() at ??:0
@ 0x7f59d1ebb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f59d1eb84de google::LogMessage::Flush() at ??:0
@ 0x7f59d1eb9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f59d4625104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f59d4622d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f59d9e121cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f59d9e1248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f59d9e12594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f59d9dd8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f59d9de2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f59d3d860d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f59d3d7b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f59d4bdf6db start_thread at ??:0
@ 0x7f59d09b661f clone at ??:0
W20260928 23:15:34.984654 16868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:34.984846 16868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:34.984869 16868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:34.986048 16868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:34.986088 16868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:34.986114 16868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:34.987511 16868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41903
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:34.987715 16868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:34.987931 16868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:34.992684 16908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:34.992748 16868 server_base.cc:1034] running on GCE node
W20260928 23:15:34.992714 16910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:34.992846 16907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:34.993125 16868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:34.993434 16868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:34.994609 16868 hybrid_clock.cc:648] HybridClock initialized: now 1790637334994585 us; error 39 us; skew 500 ppm
Sep 28 23:15:34 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:34.997478 16868 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:15:34.997884 16868 webserver.cc:460] Webserver started at http://127.0.98.1:41903/ using document root <none> and password file <none>
I20260928 23:15:34.998098 16868 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:34.998147 16868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:34.999475 16868 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:35.000118 16917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.000275 16868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.000353 16868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "83ef3357bd4740c1b984c09149aa45e5"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "d64dbe0f6e90984ded1b9185e4aa1897"
server_key_iv: "8e5346dd71cf508be3bd80b7ad29c44b"
server_key_version: "encryptionkey@0"
I20260928 23:15:35.000711 16868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:35.012621 16868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:35.014802 16868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:35.014941 16868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:35.015159 16868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:35.015594 16924 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:15:35.017557 16868 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:15:35.017601 16868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:15:35.017635 16868 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:15:35.018253 16868 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:15:35.018285 16868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 23:15:35.018385 16924 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap starting.
I20260928 23:15:35.024751 16868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42473
I20260928 23:15:35.025111 16868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:35.028481 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42473 every 8 connection(s)
I20260928 23:15:35.028825 16924 log.cc:824] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:35.031490 16924 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:35.031788 16924 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Bootstrap complete.
I20260928 23:15:35.032874 16924 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.008s
I20260928 23:15:35.033104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 16868
I20260928 23:15:35.033378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34095
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=34073
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:15:35.034287 16924 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:35.034639 16924 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Initialized, Role: FOLLOWER
I20260928 23:15:35.034801 16924 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:35.035066 16924 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
W20260928 23:15:35.082844 17034 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:15:35.025782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:41481)
0928 23:15:35.025949 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.025997 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:35.026428 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:35.082642 (+ 56214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260928 23:15:35.083098 17032 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35083: Client connection negotiation failed: client connection to 127.0.98.62:35083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 23:15:35.083441 17034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.083304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:58657)
0928 23:15:35.083354 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.083415 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:15:35.083765 17034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.083678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:51279)
0928 23:15:35.083718 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.083744 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:15:35.083892 17032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 23:15:35.112792 17036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:35.113001 17036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:35.113023 17036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:35.114214 17036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:35.114259 17036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:35.114287 17036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:35.115710 17036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34073
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:35.115927 17036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:35.116132 17036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:35.120860 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:35.120838 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:35.120937 17036 server_base.cc:1034] running on GCE node
W20260928 23:15:35.120838 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:35.121249 17036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:35.121484 17036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:35.122625 17036 hybrid_clock.cc:648] HybridClock initialized: now 1790637335122616 us; error 28 us; skew 500 ppm
Sep 28 23:15:35 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:35.124889 17036 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:15:35.125309 17036 webserver.cc:460] Webserver started at http://127.0.98.2:34073/ using document root <none> and password file <none>
I20260928 23:15:35.125501 17036 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:35.125550 17036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:35.126780 17036 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:35.127275 17052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.127426 17036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:35.127509 17036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "3f4d8a9149b54cfa9f1987e04c04ef02"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "0322442a1d20266d23845c8704cf98d2"
server_key_iv: "28a89e1c336b051e9f134c31e1501c46"
server_key_version: "encryptionkey@0"
I20260928 23:15:35.127846 17036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:35.139883 17036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:35.141534 17036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:35.141665 17036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:35.141875 17036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:35.142148 17036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:35.142179 17036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.142206 17036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:35.142228 17036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.147596 17036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34095
I20260928 23:15:35.147656 17165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34095 every 8 connection(s)
I20260928 23:15:35.147966 17036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:35.148955 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.148597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:52261)
0928 23:15:35.148749 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.148816 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
W20260928 23:15:35.149371 17166 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35083: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
I20260928 23:15:35.149606 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.149527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:56845)
0928 23:15:35.149565 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.149589 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:15:35.149814 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.149757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:51051)
0928 23:15:35.149785 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.149802 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 23:15:35.149890 17166 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:15:35.157917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17036
I20260928 23:15:35.158165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36163
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35083
--webserver_port=43427
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:35.237452 17169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:35.237653 17169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:35.237674 17169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:35.238792 17169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:35.238833 17169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:35.238862 17169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:35.240298 17169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43427
--tserver_master_addrs=127.0.98.62:35083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:35.240522 17169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:35.240732 17169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:35.245682 17174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:35.245659 17175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:35.245805 17169 server_base.cc:1034] running on GCE node
W20260928 23:15:35.245649 17177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:35.246071 17169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:35.246331 17169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:35.247469 17169 hybrid_clock.cc:648] HybridClock initialized: now 1790637335247458 us; error 24 us; skew 500 ppm
Sep 28 23:15:35 dist-test-slave-6vgd krb5kdc[15729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:35.249853 17169 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:15:35.250252 17169 webserver.cc:460] Webserver started at http://127.0.98.3:43427/ using document root <none> and password file <none>
I20260928 23:15:35.250448 17169 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:35.250492 17169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:35.251714 17169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:35.252296 17184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.252449 17169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:35.252528 17169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "2a697aff710d4da3848efab2b0e94a9e"
format_stamp: "Formatted at 2026-09-28 23:14:53 on dist-test-slave-6vgd"
server_key: "f71d8623247e791e52fdf294fffb7dfe"
server_key_iv: "a4f5bc2c94ecb7b3e98d066025e8d420"
server_key_version: "encryptionkey@0"
I20260928 23:15:35.252861 17169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:35.268671 17169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:35.270491 17169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:35.270607 17169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:35.270819 17169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:35.271268 17191 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 23:15:35.272754 17169 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 23:15:35.272817 17169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 23:15:35.272845 17169 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 23:15:35.273547 17169 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 23:15:35.273582 17169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:35.273653 17191 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Bootstrap starting.
I20260928 23:15:35.279093 17169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36163
I20260928 23:15:35.279199 17298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36163 every 8 connection(s)
I20260928 23:15:35.279485 17169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:35.280633 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.280131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:36689)
0928 23:15:35.280315 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.280429 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
W20260928 23:15:35.280843 17299 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35083 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35083: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
I20260928 23:15:35.281140 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.281033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:45839)
0928 23:15:35.281086 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.281115 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 23:15:35.281507 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.281419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:39025)
0928 23:15:35.281462 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.281487 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 23:15:35.281669 17299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 23:15:35.282371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17169
I20260928 23:15:35.283414 17191 log.cc:824] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:35.285423 17191 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:35.285635 17191 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Bootstrap complete.
I20260928 23:15:35.286326 17191 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260928 23:15:35.287467 17191 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:35.287716 17191 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a697aff710d4da3848efab2b0e94a9e, State: Initialized, Role: FOLLOWER
I20260928 23:15:35.287827 17191 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:35.288046 17191 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 23:15:35.288321 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.288221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:54761)
0928 23:15:35.288259 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.288298 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 23:15:35.288530 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:35.288480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:37189)
0928 23:15:35.288500 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:35.288518 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
I20260928 23:15:36.084451 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.084236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:49037)
0928 23:15:36.084371 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.084404 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 23:15:36.150354 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.150157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:58645)
0928 23:15:36.150270 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.150306 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 23:15:36.289088 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.288885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:50563)
0928 23:15:36.289009 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.289046 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260928 23:15:36.413829 17308 raft_consensus.cc:482] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:36.413965 17308 raft_consensus.cc:504] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:36.414275 17308 leader_election.cc:288] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163), 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
Sep 28 23:15:36 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 23:15:36 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 23:15:36.425501 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.414454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.1:60459)
0928 23:15:36.414498 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.414509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:36.414569 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:36.415382 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:36.415383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:36.415529 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:36.415736 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:36.415743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.417388 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.417389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:36.418635 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:36.418638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.419428 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.419429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:36.419570 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:36.420138 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:36.420142 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:36.421743 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:36.423693 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:36.423700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:36.423702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:36.424053 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:36.424317 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:36.424318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:36.424467 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:36.424909 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:36.424916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:36.425069 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 23:15:36.425183 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:15:36.425394 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":64}
I20260928 23:15:36.425508 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.414561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60459 (local address 127.0.98.2:34095)
0928 23:15:36.414707 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:36.414720 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:36.414761 (+ 41us) server_negotiation.cc:407] Connection header received
0928 23:15:36.415227 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:36.415229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:36.415269 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:36.415378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:36.415874 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.417296 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.418719 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.419343 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.419450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:36.421858 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:36.421863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:36.421863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:36.421876 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:36.423619 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424129 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:36.424132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:36.424133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:36.424254 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424527 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:36.424528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:36.424528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:36.424845 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:36.424926 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:36.425261 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 23:15:36.425396 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 23:15:36.425519 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.414635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46531 (local address 127.0.98.3:36163)
0928 23:15:36.414778 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:36.414789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:36.414807 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:15:36.414895 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:36.414897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:36.414943 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:36.415056 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:36.415866 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.417283 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.418725 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.419343 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.419449 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:36.422167 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:36.422170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:36.422170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:36.422180 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:36.423701 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424130 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:36.424133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:36.424133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:36.424262 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424527 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:36.424528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:36.424528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:36.424845 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:36.424926 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:36.425261 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 23:15:36.425396 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260928 23:15:36.426134 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.414529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36163 (local address 127.0.98.1:46531)
0928 23:15:36.414653 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.414661 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:36.414696 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:36.415109 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:36.415112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:36.415476 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:36.415736 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:36.415743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.417385 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.417386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:36.418635 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:36.418638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:36.419427 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:36.419428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:36.419570 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:36.420092 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:36.420103 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:36.422066 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:36.423776 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:36.423777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:36.423777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:36.424052 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:36.424317 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:36.424318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:36.424318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:36.424467 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:36.424909 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:36.424916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:36.425069 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 23:15:36.425183 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 23:15:36.425465 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":166,"thread_start_us":70,"threads_started":1}
W20260928 23:15:36.426400 16919 leader_election.cc:341] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095): Not found: Tablet not found: ba3ac82f69434465b73ddfd6b152b11a
I20260928 23:15:36.426558 17251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a697aff710d4da3848efab2b0e94a9e" is_pre_election: true
I20260928 23:15:36.426693 17251 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 1.
I20260928 23:15:36.426914 16918 leader_election.cc:302] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a697aff710d4da3848efab2b0e94a9e, 83ef3357bd4740c1b984c09149aa45e5; no voters: 3f4d8a9149b54cfa9f1987e04c04ef02
I20260928 23:15:36.427062 17308 raft_consensus.cc:2781] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 23:15:36.427107 17308 raft_consensus.cc:482] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:36.427124 17308 raft_consensus.cc:3037] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 23:15:36.428334 17308 raft_consensus.cc:504] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:36.428462 17308 leader_election.cc:288] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 election: Requested vote from peers 2a697aff710d4da3848efab2b0e94a9e (127.0.98.3:36163), 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095)
I20260928 23:15:36.428606 17251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba3ac82f69434465b73ddfd6b152b11a" candidate_uuid: "83ef3357bd4740c1b984c09149aa45e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a697aff710d4da3848efab2b0e94a9e"
I20260928 23:15:36.428685 17251 raft_consensus.cc:3037] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 1 FOLLOWER]: Advancing to term 2
W20260928 23:15:36.428746 16919 leader_election.cc:341] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095): Not found: Tablet not found: ba3ac82f69434465b73ddfd6b152b11a
I20260928 23:15:36.429819 17251 raft_consensus.cc:2454] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ef3357bd4740c1b984c09149aa45e5 in term 2.
I20260928 23:15:36.429970 16918 leader_election.cc:302] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a697aff710d4da3848efab2b0e94a9e, 83ef3357bd4740c1b984c09149aa45e5; no voters: 3f4d8a9149b54cfa9f1987e04c04ef02
I20260928 23:15:36.430071 17308 raft_consensus.cc:2781] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 23:15:36.430181 17308 raft_consensus.cc:686] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [term 2 LEADER]: Becoming Leader. State: Replica: 83ef3357bd4740c1b984c09149aa45e5, State: Running, Role: LEADER
I20260928 23:15:36.430255 17308 consensus_queue.cc:237] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:36.430464 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.430373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:43809)
0928 23:15:36.430414 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.430436 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:15:36.827020 17251 raft_consensus.cc:1261] T ba3ac82f69434465b73ddfd6b152b11a P 2a697aff710d4da3848efab2b0e94a9e [term 2 FOLLOWER]: Refusing update from remote peer 83ef3357bd4740c1b984c09149aa45e5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 23:15:36.827353 17308 consensus_queue.cc:1035] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 23:15:36.828197 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.828004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:52495)
0928 23:15:36.828112 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.828155 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":80,"threads_started":1}
I20260928 23:15:36.828892 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.828796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:37601)
0928 23:15:36.828838 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.828862 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:15:36.829087 16919 consensus_peers.cc:487] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5 -> Peer 3f4d8a9149b54cfa9f1987e04c04ef02 (127.0.98.2:34095): Couldn't send request to peer 3f4d8a9149b54cfa9f1987e04c04ef02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba3ac82f69434465b73ddfd6b152b11a. This is attempt 1: this message will repeat every 5th retry.
I20260928 23:15:36.829181 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:36.829076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:57669)
0928 23:15:36.829123 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:36.829153 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 23:15:37.150975 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.150761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:54423)
0928 23:15:37.150897 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.150935 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260928 23:15:37.402457 17322 ts_tablet_manager.cc:927] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Initiating tablet copy from peer 83ef3357bd4740c1b984c09149aa45e5 (127.0.98.1:42473)
I20260928 23:15:37.402673 17322 tablet_copy_client.cc:316] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:42473
Sep 28 23:15:37 dist-test-slave-6vgd krb5kdc[15729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 23:15:37.410178 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.402944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42473 (local address 127.0.98.2:43107)
0928 23:15:37.403001 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.403021 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:37.403086 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:37.403287 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:37.403294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:37.403564 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:37.403622 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:37.403624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:37.404537 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:37.404538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:37.405172 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:37.405176 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:37.405662 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:37.405663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:37.405729 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:37.406230 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:37.406235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:37.407558 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:37.409237 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:37.409244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:37.409246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:37.409435 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:37.409608 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:37.409609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:37.409609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:37.409657 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:37.409885 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:37.409892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:37.409939 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:15:37.410007 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:15:37.410096 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 23:15:37.410224 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.403025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43107 (local address 127.0.98.1:42473)
0928 23:15:37.403151 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:37.403156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:37.403165 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:37.403189 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:37.403191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:37.403203 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:37.403268 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:37.403700 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:37.404448 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:37.405284 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:37.405589 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:37.405650 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:37.407632 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:37.407640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:37.407641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:37.407647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:37.409148 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:37.409498 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:37.409501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:37.409502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:37.409560 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:37.409694 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:37.409695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:37.409695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:37.409828 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:37.409885 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:37.410078 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 23:15:37.410153 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":45,"threads_started":1}
I20260928 23:15:37.410629 17002 tablet_copy_service.cc:140] P 83ef3357bd4740c1b984c09149aa45e5: Received BeginTabletCopySession request for tablet ba3ac82f69434465b73ddfd6b152b11a from peer 3f4d8a9149b54cfa9f1987e04c04ef02 ({username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43107)
I20260928 23:15:37.410692 17002 tablet_copy_service.cc:161] P 83ef3357bd4740c1b984c09149aa45e5: Beginning new tablet copy session on tablet ba3ac82f69434465b73ddfd6b152b11a from peer 3f4d8a9149b54cfa9f1987e04c04ef02 at {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43107: session id = 3f4d8a9149b54cfa9f1987e04c04ef02-ba3ac82f69434465b73ddfd6b152b11a
I20260928 23:15:37.411160 17002 tablet_copy_source_session.cc:205] T ba3ac82f69434465b73ddfd6b152b11a P 83ef3357bd4740c1b984c09149aa45e5: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 23:15:37.412142 17322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba3ac82f69434465b73ddfd6b152b11a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:37.414691 17322 tablet_copy_client.cc:791] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet copy: Starting download of 0 data blocks...
I20260928 23:15:37.414836 17322 tablet_copy_client.cc:655] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet copy: Starting download of 1 WAL segments...
I20260928 23:15:37.415607 17322 tablet_copy_client.cc:531] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 23:15:37.416287 17322 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Bootstrap starting.
I20260928 23:15:37.417618 17322 log.cc:824] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:37.419941 17322 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 23:15:37.420140 17322 tablet_bootstrap.cc:492] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Bootstrap complete.
I20260928 23:15:37.420251 17322 ts_tablet_manager.cc:1397] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:15:37.421147 17322 raft_consensus.cc:348] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:37.421337 17322 raft_consensus.cc:729] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4d8a9149b54cfa9f1987e04c04ef02, State: Initialized, Role: FOLLOWER
I20260928 23:15:37.421459 17322 consensus_queue.cc:260] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a697aff710d4da3848efab2b0e94a9e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36163 } } peers { permanent_uuid: "3f4d8a9149b54cfa9f1987e04c04ef02" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "83ef3357bd4740c1b984c09149aa45e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42473 } }
I20260928 23:15:37.421689 17322 ts_tablet_manager.cc:1428] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20260928 23:15:37.421942 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.421849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:44251)
0928 23:15:37.421881 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.421912 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 23:15:37.421950 17002 tablet_copy_service.cc:342] P 83ef3357bd4740c1b984c09149aa45e5: Request end of tablet copy session 3f4d8a9149b54cfa9f1987e04c04ef02-ba3ac82f69434465b73ddfd6b152b11a received from {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43107
I20260928 23:15:37.421998 17002 tablet_copy_service.cc:433] P 83ef3357bd4740c1b984c09149aa45e5: ending tablet copy session 3f4d8a9149b54cfa9f1987e04c04ef02-ba3ac82f69434465b73ddfd6b152b11a on tablet ba3ac82f69434465b73ddfd6b152b11a with peer 3f4d8a9149b54cfa9f1987e04c04ef02
I20260928 23:15:37.422248 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.422155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:59449)
0928 23:15:37.422197 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.422224 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:15:37.829537 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.829327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:39735)
0928 23:15:37.829455 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.829491 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260928 23:15:37.829707 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.829523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:35765)
0928 23:15:37.829624 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.829658 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260928 23:15:37.938796 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.938682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:51881)
0928 23:15:37.938730 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.938759 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:15:37.938789 17120 raft_consensus.cc:1203] T ba3ac82f69434465b73ddfd6b152b11a P 3f4d8a9149b54cfa9f1987e04c04ef02 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 23:15:37.939325 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:37.939144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:38327)
0928 23:15:37.939249 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:37.939286 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260928 23:15:38.830331 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:38.830136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:54429)
0928 23:15:38.830245 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:38.830280 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 23:15:38.939419 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:38.939194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:55631)
0928 23:15:38.939330 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:38.939371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260928 23:15:38.939868 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:38.939688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:54975)
0928 23:15:38.939794 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:38.939831 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260928 23:15:39.831063 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:39.830791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:55569)
0928 23:15:39.830969 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:39.831012 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":60,"threads_started":1}
I20260928 23:15:39.940171 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:39.939931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:41455)
0928 23:15:39.940068 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:39.940124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260928 23:15:39.940538 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:39.940348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:39385)
0928 23:15:39.940467 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:39.940501 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
I20260928 23:15:40.831817 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:40.831587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:44815)
0928 23:15:40.831724 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:40.831769 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260928 23:15:40.940871 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:40.940660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:55741)
0928 23:15:40.940794 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:40.940824 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260928 23:15:40.941104 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:40.940918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:36493)
0928 23:15:40.941029 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:40.941058 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260928 23:15:40.941262 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:40.941150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:46939)
0928 23:15:40.941189 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:40.941236 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:15:41.832489 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:41.832242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:58085)
0928 23:15:41.832396 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:41.832439 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260928 23:15:41.941725 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:41.941515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:49365)
0928 23:15:41.941640 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:41.941675 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260928 23:15:41.941747 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:41.941581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:35403)
0928 23:15:41.941676 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:41.941708 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":53,"threads_started":1}
I20260928 23:15:42.833106 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:42.832906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:56161)
0928 23:15:42.833023 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:42.833059 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260928 23:15:42.942333 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:42.942134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:48085)
0928 23:15:42.942244 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:42.942285 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260928 23:15:42.942348 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:42.942114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:56303)
0928 23:15:42.942276 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:42.942306 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260928 23:15:43.833743 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:43.833532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:41627)
0928 23:15:43.833659 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:43.833693 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260928 23:15:43.942940 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:43.942749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:42831)
0928 23:15:43.942863 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:43.942894 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 23:15:43.942940 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:43.942762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:46735)
0928 23:15:43.942868 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:43.942898 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260928 23:15:44.834386 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:44.834191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:34757)
0928 23:15:44.834303 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:44.834339 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 23:15:44.943631 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:44.943400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:49719)
0928 23:15:44.943544 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:44.943584 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260928 23:15:44.943640 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:44.943400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:50159)
0928 23:15:44.943562 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:44.943598 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260928 23:15:45.835088 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:45.834850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:45421)
0928 23:15:45.834998 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:45.835035 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260928 23:15:45.944270 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:45.944068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:52401)
0928 23:15:45.944193 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:45.944226 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260928 23:15:45.944270 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:45.944068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:56615)
0928 23:15:45.944193 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:45.944228 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260928 23:15:46.835722 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:46.835489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:44871)
0928 23:15:46.835632 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:46.835674 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260928 23:15:46.944922 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:46.944700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:49095)
0928 23:15:46.944846 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:46.944880 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260928 23:15:46.944916 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:46.944694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:60091)
0928 23:15:46.944831 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:46.944867 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 23:15:47.836369 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:47.836115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:34485)
0928 23:15:47.836287 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:47.836322 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260928 23:15:47.945564 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:47.945377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:59961)
0928 23:15:47.945487 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:47.945519 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260928 23:15:47.945572 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:47.945375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:44873)
0928 23:15:47.945493 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:47.945517 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 23:15:48.836957 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:48.836776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:38181)
0928 23:15:48.836884 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:48.836913 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260928 23:15:48.946187 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:48.945965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:45897)
0928 23:15:48.946098 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:48.946139 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260928 23:15:48.946197 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:48.945990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:54161)
0928 23:15:48.946110 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:48.946143 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 23:15:49.837641 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:49.837397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.3:47081)
0928 23:15:49.837548 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:49.837586 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 23:15:49.946782 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:49.946593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.1:52091)
0928 23:15:49.946703 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:49.946736 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260928 23:15:49.946969 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:49.946593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35083 (local address 127.0.98.2:33477)
0928 23:15:49.946897 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:49.946930 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":67,"threads_started":1}
I20260928 23:15:50.292963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 16868
I20260928 23:15:50.297148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17036
I20260928 23:15:50.302484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17169
2026-09-28T23:15:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58349 ms)
[----------] 30 tests from SecurityITest (251200 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:15:50 dist-test-slave-6vgd krb5kdc[17375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:15:50 dist-test-slave-6vgd krb5kdc[17375](info): set up 2 sockets
Sep 28 23:15:50 dist-test-slave-6vgd krb5kdc[17375](info): commencing operation
krb5kdc: starting...
W20260928 23:15:51.248543 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:15:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:15:51.355891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:51.433526 17388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:51.433728 17388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:51.433749 17388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:51.434892 17388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:51.434933 17388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:51.434947 17388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:51.434957 17388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:51.434981 17388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:51.436451 17388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:51.436676 17388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:51.436882 17388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:15:51.441841 17388 server_base.cc:1034] running on GCE node
W20260928 23:15:51.441920 17393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.441886 17394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.441954 17396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:51.442343 17388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:51.442638 17388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:51.443785 17388 hybrid_clock.cc:648] HybridClock initialized: now 1790637351443766 us; error 32 us; skew 500 ppm
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:51.446095 17388 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:15:51.446512 17388 webserver.cc:460] Webserver started at http://127.0.98.62:42327/ using document root <none> and password file <none>
I20260928 23:15:51.446709 17388 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:51.446755 17388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:51.446872 17388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:51.447873 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "c0ae836a60ac4d7e8f869356394ccf9c"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "b0d7ee25ade535923104434a6ecea6e5"
server_key_iv: "cd54d7bddd7a070715005097ed44d1e3"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.448199 17388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0ae836a60ac4d7e8f869356394ccf9c"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "b0d7ee25ade535923104434a6ecea6e5"
server_key_iv: "cd54d7bddd7a070715005097ed44d1e3"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.449419 17388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:51.450078 17403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.450217 17388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:51.450322 17388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "c0ae836a60ac4d7e8f869356394ccf9c"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "b0d7ee25ade535923104434a6ecea6e5"
server_key_iv: "cd54d7bddd7a070715005097ed44d1e3"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.450382 17388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:51.467860 17388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:51.469615 17388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:51.469753 17388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:51.473551 17388 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37415
I20260928 23:15:51.473592 17455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37415 every 8 connection(s)
I20260928 23:15:51.473917 17388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:51.474697 17456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:51.476822 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: Bootstrap starting.
I20260928 23:15:51.477367 17456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:51.477588 17456 log.cc:824] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:51.478159 17456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: No bootstrap required, opened a new log
I20260928 23:15:51.479660 17456 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:15:51.479794 17456 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:51.479821 17456 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ae836a60ac4d7e8f869356394ccf9c, State: Initialized, Role: FOLLOWER
I20260928 23:15:51.479996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17388
I20260928 23:15:51.479977 17456 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:15:51.480064 17456 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:51.480094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:51.480103 17456 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:51.480144 17456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:51.480333 392 external_mini_cluster.cc:1428] Setting key 9afdc40f87cf1fb81b2e696044e48ccf
I20260928 23:15:51.480768 17456 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:15:51.480877 17456 leader_election.cc:302] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ae836a60ac4d7e8f869356394ccf9c; no voters:
I20260928 23:15:51.481038 17456 leader_election.cc:288] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:51.481069 17461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:51.481240 17461 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [term 1 LEADER]: Becoming Leader. State: Replica: c0ae836a60ac4d7e8f869356394ccf9c, State: Running, Role: LEADER
I20260928 23:15:51.481292 17456 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:51.481381 17461 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } }
I20260928 23:15:51.481952 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260928 23:15:51.482043 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:51.482489 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ae836a60ac4d7e8f869356394ccf9c. Latest consensus state: current_term: 1 leader_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae836a60ac4d7e8f869356394ccf9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37415 } } }
I20260928 23:15:51.482566 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:51.482656 17470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:51.483238 17470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:51.484763 17470 catalog_manager.cc:1347] Generated new cluster ID: e019cb8b9e704940a3c2f6c44eeaebe6
I20260928 23:15:51.484819 17470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:51.490342 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.480802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.62:37415)
0928 23:15:51.480962 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.480972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.481009 (+ 37us) server_negotiation.cc:407] Connection header received
0928 23:15:51.481130 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.481138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.481200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.482122 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:51.482252 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.483779 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.484587 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.485167 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.485327 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.487437 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:51.487441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:51.487442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:51.487454 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:51.488931 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.489282 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.489285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.489286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.489398 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.489574 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.489575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.489575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.489806 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:51.489963 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.490033 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.490166 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260928 23:15:51.503985 17470 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:51.504724 17470 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:51.511498 17470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: Generated new TSK 0
I20260928 23:15:51.511677 17470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:15:51.534232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:51.611840 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:51.612040 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:51.612069 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:51.613189 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:51.613258 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:51.613289 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:51.614722 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:51.614929 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:51.615134 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:51.619688 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.619721 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.619725 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:51.619871 17483 server_base.cc:1034] running on GCE node
I20260928 23:15:51.620067 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:51.620282 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:51.621420 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790637351621408 us; error 28 us; skew 500 ppm
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:51.623649 17483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:15:51.624051 17483 webserver.cc:460] Webserver started at http://127.0.98.1:39333/ using document root <none> and password file <none>
I20260928 23:15:51.624234 17483 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:51.624279 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:51.624404 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:51.625407 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5a508818246454aa6e6d5d7f39bab1f"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "5915a42283ee9de72390749763afdd33"
server_key_iv: "826c015c3b4775f895ff8dd8334e2dee"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.625712 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5a508818246454aa6e6d5d7f39bab1f"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "5915a42283ee9de72390749763afdd33"
server_key_iv: "826c015c3b4775f895ff8dd8334e2dee"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.626837 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:51.627413 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.627547 17483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:51.627622 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "f5a508818246454aa6e6d5d7f39bab1f"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "5915a42283ee9de72390749763afdd33"
server_key_iv: "826c015c3b4775f895ff8dd8334e2dee"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.627676 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:51.643035 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:51.644681 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:51.644807 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:51.645007 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:51.645359 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:51.645409 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.645438 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:51.645463 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.650839 17483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38353
I20260928 23:15:51.650878 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38353 every 8 connection(s)
I20260928 23:15:51.651168 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:51.658638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17483
I20260928 23:15:51.658732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:51.658903 392 external_mini_cluster.cc:1428] Setting key 733f8e08a9c4b7cd09ba5ebd4985f719
I20260928 23:15:51.660174 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.651791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34561 (local address 127.0.98.62:37415)
0928 23:15:51.651860 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.651862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.652045 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:15:51.652995 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.652996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.653008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.653058 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:51.653693 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.654387 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.655534 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.655726 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.655775 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.657743 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:51.657745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:51.657745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:51.657750 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:51.658887 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.659297 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.659298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.659299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.659358 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.659570 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.659571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.659571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.659678 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:51.659731 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.659989 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.660072 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:15:51.660346 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.651774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.1:34561)
0928 23:15:51.651927 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.651967 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.652911 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.653122 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.653125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.653424 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:51.653616 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.653623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.654474 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.654475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.655464 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.655467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.655783 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.655783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.655879 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.656353 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:51.656361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:51.657662 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:51.658954 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.658961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.658963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.659232 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.659411 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.659412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.659412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.659517 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.659730 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:51.659737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:51.659871 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.659960 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.660113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 23:15:51.660806 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:15:51.660910 17612 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:51.661094 17612 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:15:51.661643 17420 ts_manager.cc:194] Registered new tserver with Master: f5a508818246454aa6e6d5d7f39bab1f (127.0.98.1:38353)
I20260928 23:15:51.662487 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34561
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:15:51.679652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:51.756913 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:51.757091 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:51.757112 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:51.758230 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:51.758275 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:51.758306 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:51.759739 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:51.759944 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:51.760154 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:51.764694 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.764735 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:51.764837 17617 server_base.cc:1034] running on GCE node
W20260928 23:15:51.764926 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:51.765126 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:51.765357 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:51.766494 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790637351766481 us; error 31 us; skew 500 ppm
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:51.768796 17617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:15:51.769177 17617 webserver.cc:460] Webserver started at http://127.0.98.2:37455/ using document root <none> and password file <none>
I20260928 23:15:51.769392 17617 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:51.769440 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:51.769558 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:51.770558 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "a2223760f77544d3b52cb707d9927082"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "c60f2ef72bb5c6ae03656a2202262a1b"
server_key_iv: "13567fb91d91224a66f0ef836688a3c7"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.770881 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a2223760f77544d3b52cb707d9927082"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "c60f2ef72bb5c6ae03656a2202262a1b"
server_key_iv: "13567fb91d91224a66f0ef836688a3c7"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.772047 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:15:51.772643 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.772790 17617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:15:51.772867 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "a2223760f77544d3b52cb707d9927082"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "c60f2ef72bb5c6ae03656a2202262a1b"
server_key_iv: "13567fb91d91224a66f0ef836688a3c7"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.772925 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:51.790544 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:51.792079 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:51.792201 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:51.792388 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:51.792649 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:51.792680 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.792706 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:51.792723 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.797878 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40659
I20260928 23:15:51.797914 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40659 every 8 connection(s)
I20260928 23:15:51.798190 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:51.803606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17617
I20260928 23:15:51.803701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:51.803884 392 external_mini_cluster.cc:1428] Setting key ec2504dd019fec84294f4008280c0031
I20260928 23:15:51.806479 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.798835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58621 (local address 127.0.98.62:37415)
0928 23:15:51.798902 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.798903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.799060 (+ 157us) server_negotiation.cc:407] Connection header received
0928 23:15:51.799925 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.799926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.799936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.799986 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:51.800553 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.801042 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.801957 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.802146 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.802193 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.804038 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:51.804039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:51.804039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:51.804047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:51.805316 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.805665 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.805666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.805666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.805715 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.805919 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.805920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.805920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.806021 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:51.806074 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.806335 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.806414 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:15:51.806653 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.798800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.2:58621)
0928 23:15:51.798952 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.798985 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.799830 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.800040 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.800043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.800294 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:51.800472 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.800476 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.801103 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.801104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.801896 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.801898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.802200 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.802201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.802289 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.802675 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:51.802682 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:51.803960 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:51.805379 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.805386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.805388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.805594 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.805765 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.805766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.805766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.805867 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.806072 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:51.806078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:51.806206 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.806295 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.806438 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260928 23:15:51.807080 17746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:15:51.807161 17746 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:51.807315 17746 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:15:51.807672 17420 ts_manager.cc:194] Registered new tserver with Master: a2223760f77544d3b52cb707d9927082 (127.0.98.2:40659)
I20260928 23:15:51.808024 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58621
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:15:51.825502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:51.902534 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:51.902742 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:51.902765 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:51.903911 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:51.903951 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:51.903980 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:51.905450 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:51.905663 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:51.905882 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:51.910535 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.910535 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:51.910542 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:51.910615 17751 server_base.cc:1034] running on GCE node
I20260928 23:15:51.910974 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:51.911197 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:51.912328 17751 hybrid_clock.cc:648] HybridClock initialized: now 1790637351912315 us; error 28 us; skew 500 ppm
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:51.914587 17751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:15:51.914987 17751 webserver.cc:460] Webserver started at http://127.0.98.3:37769/ using document root <none> and password file <none>
I20260928 23:15:51.915169 17751 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:51.915215 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:51.915334 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:51.916311 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb19bee44d104f488e8266542ef42045"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "05e02608fdd42543fe7a7592ed216d46"
server_key_iv: "ce6830ec6ce60cd67313739f447f3558"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.916625 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb19bee44d104f488e8266542ef42045"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "05e02608fdd42543fe7a7592ed216d46"
server_key_iv: "ce6830ec6ce60cd67313739f447f3558"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.917819 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 23:15:51.918496 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.918633 17751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:51.918717 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "bb19bee44d104f488e8266542ef42045"
format_stamp: "Formatted at 2026-09-28 23:15:51 on dist-test-slave-6vgd"
server_key: "05e02608fdd42543fe7a7592ed216d46"
server_key_iv: "ce6830ec6ce60cd67313739f447f3558"
server_key_version: "encryptionkey@0"
I20260928 23:15:51.918776 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:51.938985 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:51.940798 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:51.940938 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:51.941149 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:51.941483 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:51.941516 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.941545 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:51.941567 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:51.946875 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44289
I20260928 23:15:51.946929 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44289 every 8 connection(s)
I20260928 23:15:51.947207 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:51.949424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17751
I20260928 23:15:51.949538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:51.949712 392 external_mini_cluster.cc:1428] Setting key 2fca0c22d7fe0f69d4505fb8c70b476c
Sep 28 23:15:51 dist-test-slave-6vgd krb5kdc[17375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:51.956421 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.947822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45357 (local address 127.0.98.62:37415)
0928 23:15:51.947879 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.947881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.948071 (+ 190us) server_negotiation.cc:407] Connection header received
0928 23:15:51.949068 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.949069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.949083 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.949130 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:51.949935 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.950658 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.952057 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.952246 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.952299 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.954239 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:51.954240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:51.954241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:51.954249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:51.955263 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.955593 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.955594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.955594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.955643 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.955849 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.955850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.955850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.955951 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:51.956005 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.956258 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.956334 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 23:15:51.956573 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.947807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37415 (local address 127.0.98.3:45357)
0928 23:15:51.947957 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.947992 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.948966 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.949227 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.949231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.949574 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:51.949829 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.949834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.950771 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.950772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.951982 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.951985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.952310 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.952311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.952437 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.952829 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:51.952837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:51.954145 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:51.955329 (+ 1184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.955335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.955338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.955532 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.955696 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:51.955696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:51.955697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:51.955797 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:51.956003 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:51.956009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:51.956138 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.956229 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.956370 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260928 23:15:51.956964 17880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37415
I20260928 23:15:51.957046 17880 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:51.957240 17880 heartbeater.cc:511] Master 127.0.98.62:37415 requested a full tablet report, sending...
I20260928 23:15:51.957578 17420 ts_manager.cc:194] Registered new tserver with Master: bb19bee44d104f488e8266542ef42045 (127.0.98.3:44289)
I20260928 23:15:51.957906 17420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45357
I20260928 23:15:51.961864 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:51.967833 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.963052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.98.62:37415)
0928 23:15:51.963097 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.963098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.963203 (+ 105us) server_negotiation.cc:407] Connection header received
0928 23:15:51.963254 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.963255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.963263 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.963298 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:51.963773 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.964219 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.964861 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.965027 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.965066 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.966024 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:51.966025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:51.966025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:51.966031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:51.967037 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.967304 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.967305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.967305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.967344 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:51.967459 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:51.967459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:51.967459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:51.967574 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:51.967638 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.967715 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.967776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:15:51.969038 17420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:51.970571 17420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:51.976150 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44289 (local address 127.0.0.1:60166)
0928 23:15:51.972768 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.972778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.972839 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.973079 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.973082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.973083 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:51.973162 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.973164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.974064 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.974064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.975105 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.975107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.975862 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.975863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.975941 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.975955 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.976027 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.976074 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":60,"threads_started":1}
I20260928 23:15:51.976209 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.0.98.3:44289)
0928 23:15:51.972753 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.972758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.972835 (+ 77us) server_negotiation.cc:407] Connection header received
0928 23:15:51.972904 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.972906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.972922 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.972991 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:51.973259 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.973978 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.975177 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.975782 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.975852 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.975911 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.976044 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.976082 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 23:15:51.976660 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40659 (local address 127.0.0.1:54112)
0928 23:15:51.972858 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.972867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.972903 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.973265 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.973266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.973268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:51.973310 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.973312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.974588 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.974589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.975574 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.975576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.976447 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.976448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.976512 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.976517 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.976558 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.976598 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":60,"threads_started":1}
I20260928 23:15:51.976992 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.0.98.2:40659)
0928 23:15:51.973017 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.973021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.973028 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:15:51.973064 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.973066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.973081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.973154 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:51.973385 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.974492 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.975639 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.976326 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.976793 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.976860 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.976884 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.976921 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260928 23:15:51.977108 17814 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 (DEFAULT_TABLE table=test-table [id=e3f96dd0bef541abbd409341fea01fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:51.977474 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ad7b3b36d4b91baf2f4a995fabd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:51.977809 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38353 (local address 127.0.0.1:46210)
0928 23:15:51.972635 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:51.972674 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:51.972737 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:51.973267 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:51.973268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:51.973270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:51.973316 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.973317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.975639 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.975640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.976551 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:51.976553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.977508 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.977509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:51.977666 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.977671 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:51.977707 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 23:15:51.977741 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":10,"thread_start_us":54,"threads_started":1}
I20260928 23:15:51.977811 17680 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 (DEFAULT_TABLE table=test-table [id=e3f96dd0bef541abbd409341fea01fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:51.978096 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ad7b3b36d4b91baf2f4a995fabd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:51.979900 17894 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Bootstrap starting.
I20260928 23:15:51.980489 17895 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Bootstrap starting.
I20260928 23:15:51.980620 17894 tablet_bootstrap.cc:654] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:51.980679 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:51.972589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.0.98.1:38353)
0928 23:15:51.973014 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:51.973019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:51.973028 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:51.973064 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:51.973066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:51.973081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:51.973154 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:51.973385 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.974486 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.976632 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:51.977302 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:51.977367 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:51.980530 (+ 3163us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:51.980556 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 23:15:51.980603 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":90,"threads_started":1}
I20260928 23:15:51.981077 17895 tablet_bootstrap.cc:654] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:51.981115 17894 log.cc:824] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:51.981379 17546 tablet_service.cc:1467] Processing CreateTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 (DEFAULT_TABLE table=test-table [id=e3f96dd0bef541abbd409341fea01fc5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:51.981603 17895 log.cc:824] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:51.981714 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0ad7b3b36d4b91baf2f4a995fabd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:51.981879 17894 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: No bootstrap required, opened a new log
I20260928 23:15:51.981951 17894 ts_tablet_manager.cc:1397] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:15:51.982292 17895 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: No bootstrap required, opened a new log
I20260928 23:15:51.982358 17895 ts_tablet_manager.cc:1397] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:15:51.983485 17894 raft_consensus.cc:348] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.983639 17894 raft_consensus.cc:374] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:51.983678 17894 raft_consensus.cc:729] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb19bee44d104f488e8266542ef42045, State: Initialized, Role: FOLLOWER
I20260928 23:15:51.983805 17894 consensus_queue.cc:260] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.984083 17894 ts_tablet_manager.cc:1428] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:15:51.984205 17880 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
I20260928 23:15:51.984323 17895 raft_consensus.cc:348] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.984496 17895 raft_consensus.cc:374] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:51.984535 17895 raft_consensus.cc:729] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2223760f77544d3b52cb707d9927082, State: Initialized, Role: FOLLOWER
I20260928 23:15:51.984647 17895 consensus_queue.cc:260] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.984894 17895 ts_tablet_manager.cc:1428] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 23:15:51.984906 17899 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: Bootstrap starting.
I20260928 23:15:51.985001 17746 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
I20260928 23:15:51.985567 17899 tablet_bootstrap.cc:654] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:51.985980 17899 log.cc:824] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:51.986519 17899 tablet_bootstrap.cc:492] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: No bootstrap required, opened a new log
I20260928 23:15:51.986574 17899 ts_tablet_manager.cc:1397] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:51.987872 17899 raft_consensus.cc:348] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.988036 17899 raft_consensus.cc:374] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:51.988077 17899 raft_consensus.cc:729] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5a508818246454aa6e6d5d7f39bab1f, State: Initialized, Role: FOLLOWER
I20260928 23:15:51.988201 17899 consensus_queue.cc:260] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:51.988413 17899 ts_tablet_manager.cc:1428] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:51.988445 17612 heartbeater.cc:503] Master 127.0.98.62:37415 was elected leader, sending a full tablet report...
I20260928 23:15:52.046247 17898 raft_consensus.cc:482] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:52.046358 17898 raft_consensus.cc:504] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:52.046684 17898 leader_election.cc:288] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5a508818246454aa6e6d5d7f39bab1f (127.0.98.1:38353), a2223760f77544d3b52cb707d9927082 (127.0.98.2:40659)
W20260928 23:15:52.048668 17747 tablet.cc:2369] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:52.050316 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.046942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49843 (local address 127.0.98.1:38353)
0928 23:15:52.047008 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.047010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.047019 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:52.047047 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.047047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.047060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.047113 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.047272 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.048020 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.049415 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.049974 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.050041 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.050050 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.050209 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.050248 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 23:15:52.050315 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.047034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34961 (local address 127.0.98.2:40659)
0928 23:15:52.047074 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.047076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.047169 (+ 93us) server_negotiation.cc:407] Connection header received
0928 23:15:52.047228 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.047230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.047241 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.047290 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.047392 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.048060 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.049405 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.049974 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.050041 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.050050 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.050216 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.050251 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:15:52.050334 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.046845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38353 (local address 127.0.98.3:49843)
0928 23:15:52.046904 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.046916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.046967 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.047122 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.047124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.047126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.047196 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.047197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.048092 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.048093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.049329 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.049332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.050053 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.050054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.050146 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.050148 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.050196 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.050243 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":13}
I20260928 23:15:52.050341 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.046937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40659 (local address 127.0.98.3:34961)
0928 23:15:52.047102 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.047109 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.047176 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.047297 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.047298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.047299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.047341 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.047342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.048137 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.048137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.049330 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.049332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.050053 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.050054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.050144 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.050148 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.050196 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.050243 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":8,"thread_start_us":106,"threads_started":1}
I20260928 23:15:52.050654 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ad7b3b36d4b91baf2f4a995fabd72" candidate_uuid: "bb19bee44d104f488e8266542ef42045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a508818246454aa6e6d5d7f39bab1f" is_pre_election: true
I20260928 23:15:52.050800 17566 raft_consensus.cc:2454] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb19bee44d104f488e8266542ef42045 in term 0.
I20260928 23:15:52.051012 17767 leader_election.cc:302] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb19bee44d104f488e8266542ef42045, f5a508818246454aa6e6d5d7f39bab1f; no voters:
I20260928 23:15:52.051080 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ad7b3b36d4b91baf2f4a995fabd72" candidate_uuid: "bb19bee44d104f488e8266542ef42045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2223760f77544d3b52cb707d9927082" is_pre_election: true
I20260928 23:15:52.051177 17898 raft_consensus.cc:2781] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:52.051214 17700 raft_consensus.cc:2454] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb19bee44d104f488e8266542ef42045 in term 0.
I20260928 23:15:52.051229 17898 raft_consensus.cc:482] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:52.051251 17898 raft_consensus.cc:3037] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.051925 17898 raft_consensus.cc:504] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:52.052028 17898 leader_election.cc:288] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [CANDIDATE]: Term 1 election: Requested vote from peers f5a508818246454aa6e6d5d7f39bab1f (127.0.98.1:38353), a2223760f77544d3b52cb707d9927082 (127.0.98.2:40659)
I20260928 23:15:52.052217 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ad7b3b36d4b91baf2f4a995fabd72" candidate_uuid: "bb19bee44d104f488e8266542ef42045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2223760f77544d3b52cb707d9927082"
I20260928 23:15:52.052219 17566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0ad7b3b36d4b91baf2f4a995fabd72" candidate_uuid: "bb19bee44d104f488e8266542ef42045" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5a508818246454aa6e6d5d7f39bab1f"
I20260928 23:15:52.052275 17700 raft_consensus.cc:3037] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.052304 17566 raft_consensus.cc:3037] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.052971 17700 raft_consensus.cc:2454] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb19bee44d104f488e8266542ef42045 in term 1.
I20260928 23:15:52.053094 17566 raft_consensus.cc:2454] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb19bee44d104f488e8266542ef42045 in term 1.
I20260928 23:15:52.053115 17767 leader_election.cc:302] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2223760f77544d3b52cb707d9927082, bb19bee44d104f488e8266542ef42045; no voters:
I20260928 23:15:52.053187 17898 raft_consensus.cc:2781] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:52.053351 17898 raft_consensus.cc:686] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 1 LEADER]: Becoming Leader. State: Replica: bb19bee44d104f488e8266542ef42045, State: Running, Role: LEADER
I20260928 23:15:52.053436 17898 consensus_queue.cc:237] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } }
I20260928 23:15:52.054194 17420 catalog_manager.cc:5495] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb19bee44d104f488e8266542ef42045 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bb19bee44d104f488e8266542ef42045" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb19bee44d104f488e8266542ef42045" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 } health_report { overall_health: UNKNOWN } } }
I20260928 23:15:52.066812 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.064796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.0.98.3:44289)
0928 23:15:52.064839 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.064841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.064850 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:52.064898 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.064899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.064915 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.064970 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:52.065062 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.065557 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.066230 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.066406 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.066448 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.066552 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:52.066655 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:52.066711 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.066723 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.066739 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:15:52.068506 17566 raft_consensus.cc:1261] T ed0ad7b3b36d4b91baf2f4a995fabd72 P f5a508818246454aa6e6d5d7f39bab1f [term 1 FOLLOWER]: Refusing update from remote peer bb19bee44d104f488e8266542ef42045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:52.068506 17700 raft_consensus.cc:1261] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 1 FOLLOWER]: Refusing update from remote peer bb19bee44d104f488e8266542ef42045: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:52.068861 17904 consensus_queue.cc:1035] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2223760f77544d3b52cb707d9927082" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:52.068988 17898 consensus_queue.cc:1035] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5a508818246454aa6e6d5d7f39bab1f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:52.071792 17908 mvcc.cc:205] Tried to move back new op lower bound from 7334450594070335488 to 7334450594011774976. Current Snapshot: MvccSnapshot[applied={T|T < 7334450594070335488}]
I20260928 23:15:52.071938 17909 mvcc.cc:205] Tried to move back new op lower bound from 7334450594070335488 to 7334450594011774976. Current Snapshot: MvccSnapshot[applied={T|T < 7334450594070335488}]
I20260928 23:15:52.073786 17419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:52.073865 17419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:52.074543 17419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0ae836a60ac4d7e8f869356394ccf9c: Sending DeleteTablet for 3 replicas of tablet ed0ad7b3b36d4b91baf2f4a995fabd72
I20260928 23:15:52.074862 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46210
I20260928 23:15:52.074973 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60166
I20260928 23:15:52.075143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17483
I20260928 23:15:52.075242 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0ad7b3b36d4b91baf2f4a995fabd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54112
I20260928 23:15:52.076274 17919 tablet_replica.cc:330] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: stopping tablet replica
I20260928 23:15:52.076403 17919 raft_consensus.cc:2229] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:52.076536 17919 raft_consensus.cc:2258] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:52.076711 17918 tablet_replica.cc:330] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: stopping tablet replica
I20260928 23:15:52.076856 17918 raft_consensus.cc:2229] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:52.076972 17918 raft_consensus.cc:2258] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:52.077351 17918 ts_tablet_manager.cc:1905] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:52.077659 17919 ts_tablet_manager.cc:1905] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:52.079059 17918 ts_tablet_manager.cc:1918] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:52.079128 17918 log.cc:1196] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/ed0ad7b3b36d4b91baf2f4a995fabd72
I20260928 23:15:52.079313 17919 ts_tablet_manager.cc:1918] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:52.079377 17918 ts_tablet_manager.cc:1939] T ed0ad7b3b36d4b91baf2f4a995fabd72 P bb19bee44d104f488e8266542ef42045: Deleting consensus metadata
I20260928 23:15:52.079473 17919 log.cc:1196] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/ed0ad7b3b36d4b91baf2f4a995fabd72
I20260928 23:15:52.079746 17406 catalog_manager.cc:4841] TS bb19bee44d104f488e8266542ef42045 (127.0.98.3:44289): tablet ed0ad7b3b36d4b91baf2f4a995fabd72 (table test-table [id=e3f96dd0bef541abbd409341fea01fc5]) successfully deleted
I20260928 23:15:52.079783 17919 ts_tablet_manager.cc:1939] T ed0ad7b3b36d4b91baf2f4a995fabd72 P a2223760f77544d3b52cb707d9927082: Deleting consensus metadata
I20260928 23:15:52.080243 17404 catalog_manager.cc:4841] TS a2223760f77544d3b52cb707d9927082 (127.0.98.2:40659): tablet ed0ad7b3b36d4b91baf2f4a995fabd72 (table test-table [id=e3f96dd0bef541abbd409341fea01fc5]) successfully deleted
W20260928 23:15:52.080909 17404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38353 (error 108)
I20260928 23:15:52.081110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17617
I20260928 23:15:52.082585 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.082465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38353 (local address 127.0.0.1:46214)
0928 23:15:52.082516 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.082553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 23:15:52.082700 17404 catalog_manager.cc:4584] TS f5a508818246454aa6e6d5d7f39bab1f (127.0.98.1:38353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed0ad7b3b36d4b91baf2f4a995fabd72: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38353: connect: Connection refused (error 111)
I20260928 23:15:52.085664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17751
I20260928 23:15:52.089537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17388
2026-09-28T23:15:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T23:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:52Z Disabled control of system clock
I20260928 23:15:52.114408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33199
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:52.190817 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:52.191020 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:52.191037 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:52.192097 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:52.192134 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:52.192147 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:52.192157 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:52.192170 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:52.193434 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:52.193630 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:52.193907 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:52.198318 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:52.198315 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.198446 17924 server_base.cc:1034] running on GCE node
W20260928 23:15:52.198455 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.198877 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:52.199149 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:52.200276 17924 hybrid_clock.cc:648] HybridClock initialized: now 1790637352200264 us; error 34 us; skew 500 ppm
I20260928 23:15:52.201409 17924 webserver.cc:460] Webserver started at http://127.0.98.62:35075/ using document root <none> and password file <none>
I20260928 23:15:52.201627 17924 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:52.201683 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:52.201807 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:52.202888 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "a2fbc6b6c3404b5bb1265ff8137d5272"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "fa79b1b2af5a7a0aa04250e5711002a6"
server_key_iv: "d22e60647df58e7292e95e4a8a5d02de"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.203210 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2fbc6b6c3404b5bb1265ff8137d5272"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "fa79b1b2af5a7a0aa04250e5711002a6"
server_key_iv: "d22e60647df58e7292e95e4a8a5d02de"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.204339 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:52.204942 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.205085 17924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:52.205168 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "a2fbc6b6c3404b5bb1265ff8137d5272"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "fa79b1b2af5a7a0aa04250e5711002a6"
server_key_iv: "d22e60647df58e7292e95e4a8a5d02de"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.205247 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:52.226759 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:52.227032 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:52.227129 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:52.230607 17924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33199
I20260928 23:15:52.230648 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33199 every 8 connection(s)
I20260928 23:15:52.230948 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:52.231846 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:52.234016 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: Bootstrap starting.
I20260928 23:15:52.234561 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:52.234818 17991 log.cc:824] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:52.235409 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: No bootstrap required, opened a new log
I20260928 23:15:52.236908 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } }
I20260928 23:15:52.237038 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:52.237071 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fbc6b6c3404b5bb1265ff8137d5272, State: Initialized, Role: FOLLOWER
I20260928 23:15:52.237243 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } }
I20260928 23:15:52.237320 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:52.237350 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:52.237396 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.237821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 17924
I20260928 23:15:52.237860 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } }
I20260928 23:15:52.237902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:52.237946 17991 leader_election.cc:302] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2fbc6b6c3404b5bb1265ff8137d5272; no voters:
I20260928 23:15:52.238085 392 external_mini_cluster.cc:1428] Setting key d0539b98857050208a687acf5b3a288c
I20260928 23:15:52.238106 17991 leader_election.cc:288] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:52.238145 17994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:52.238271 17994 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [term 1 LEADER]: Becoming Leader. State: Replica: a2fbc6b6c3404b5bb1265ff8137d5272, State: Running, Role: LEADER
I20260928 23:15:52.238339 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:52.238365 17994 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } }
I20260928 23:15:52.238688 17994 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2fbc6b6c3404b5bb1265ff8137d5272. Latest consensus state: current_term: 1 leader_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } } }
I20260928 23:15:52.238756 17994 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:52.238935 18002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:52.239034 17995 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fbc6b6c3404b5bb1265ff8137d5272" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33199 } } }
I20260928 23:15:52.239249 17995 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:52.239425 18002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:52.240857 18002 catalog_manager.cc:1347] Generated new cluster ID: 7b8f84d75cb14a7b8f661e4a35434fca
I20260928 23:15:52.240906 18002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:52.244616 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.238676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:33199)
0928 23:15:52.239460 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.239469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.239502 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:15:52.239697 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.239704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.239772 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.239958 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:52.240214 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.241525 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.242560 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.243161 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.243281 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.243871 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:52.243875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:52.243875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:52.244086 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:52.244127 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:52.244142 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:52.244209 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.244247 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.244417 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":86,"threads_started":1}
I20260928 23:15:52.249162 18002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:52.249837 18002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:52.258353 18002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: Generated new TSK 0
I20260928 23:15:52.258539 18002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:15:52.261888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--builtin_ntp_servers=127.0.98.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:52.340519 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:52.340730 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:52.340760 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:52.341995 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:52.342038 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:52.342073 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:52.343411 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:52.343633 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:52.343899 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:52.349517 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.349604 18015 server_base.cc:1034] running on GCE node
W20260928 23:15:52.349520 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:52.349727 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.349961 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:52.350195 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:52.351339 18015 hybrid_clock.cc:648] HybridClock initialized: now 1790637352351328 us; error 31 us; skew 500 ppm
I20260928 23:15:52.352545 18015 webserver.cc:460] Webserver started at http://127.0.98.1:36057/ using document root <none> and password file <none>
I20260928 23:15:52.352766 18015 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:52.352807 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:52.352948 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:52.354056 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc55b9cc715a470fab493e8ca5b3cff8"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "5aa9852bf8674e100ecb849d468d4e5d"
server_key_iv: "ab5057738ca8efbb05f8bff4a952392d"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.354364 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc55b9cc715a470fab493e8ca5b3cff8"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "5aa9852bf8674e100ecb849d468d4e5d"
server_key_iv: "ab5057738ca8efbb05f8bff4a952392d"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.355554 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:15:52.356230 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.356374 18015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:15:52.356447 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "cc55b9cc715a470fab493e8ca5b3cff8"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "5aa9852bf8674e100ecb849d468d4e5d"
server_key_iv: "ab5057738ca8efbb05f8bff4a952392d"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.356514 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:52.366416 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:52.366668 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:52.366787 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:52.366984 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:52.367261 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:52.367292 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.367319 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:52.367343 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.372692 18015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33945
I20260928 23:15:52.372730 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33945 every 8 connection(s)
I20260928 23:15:52.373029 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:52.376613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18015
I20260928 23:15:52.376719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:52.376921 392 external_mini_cluster.cc:1428] Setting key 7083af01d24d643a24e1aeb76ca76477
I20260928 23:15:52.377823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--builtin_ntp_servers=127.0.98.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:15:52.378490 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.373692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47479 (local address 127.0.98.62:33199)
0928 23:15:52.373768 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.373770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.373950 (+ 180us) server_negotiation.cc:407] Connection header received
0928 23:15:52.374534 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.374535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.374546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.374590 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:52.375379 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.375878 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.377051 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.377310 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.377371 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.378239 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:52.378240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:52.378240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:52.378244 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:52.378260 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:52.378264 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:52.378309 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.378383 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.378414 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:15:52.378695 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.373673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33199 (local address 127.0.98.1:47479)
0928 23:15:52.373837 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.373871 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.374446 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.374654 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.374657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.374665 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:52.375291 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.375297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.375960 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.375961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.376973 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.376975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.377388 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.377389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.377547 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.378108 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:52.378108 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:52.378138 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:52.378161 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:52.378320 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:52.378324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:52.378328 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.378386 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.378493 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260928 23:15:52.379140 18143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33199
I20260928 23:15:52.379222 18143 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:52.379390 18143 heartbeater.cc:511] Master 127.0.98.62:33199 requested a full tablet report, sending...
I20260928 23:15:52.379901 17955 ts_manager.cc:194] Registered new tserver with Master: cc55b9cc715a470fab493e8ca5b3cff8 (127.0.98.1:33945)
I20260928 23:15:52.380482 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47479
W20260928 23:15:52.455447 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:52.455641 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:52.455662 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:52.456794 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:52.456833 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:52.456859 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:52.458199 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:52.458427 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:52.458693 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:52.463459 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.463529 18146 server_base.cc:1034] running on GCE node
W20260928 23:15:52.463462 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:52.463459 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.463910 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:52.464160 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:52.465271 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790637352465251 us; error 33 us; skew 500 ppm
I20260928 23:15:52.466367 18146 webserver.cc:460] Webserver started at http://127.0.98.2:33681/ using document root <none> and password file <none>
I20260928 23:15:52.466555 18146 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:52.466591 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:52.466679 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:52.467658 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "6578651d0bb94d308827f17052d4709b"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "1a1724fdfe83216ecf7a9fd0c6ed3e3b"
server_key_iv: "01bb5819032e41979766b6a7df0d9955"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.467940 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6578651d0bb94d308827f17052d4709b"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "1a1724fdfe83216ecf7a9fd0c6ed3e3b"
server_key_iv: "01bb5819032e41979766b6a7df0d9955"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.469048 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:15:52.469772 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.469947 18146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 23:15:52.470026 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "6578651d0bb94d308827f17052d4709b"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "1a1724fdfe83216ecf7a9fd0c6ed3e3b"
server_key_iv: "01bb5819032e41979766b6a7df0d9955"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.470077 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:52.476845 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:52.477069 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:52.477160 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:52.477360 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:52.477626 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:52.477669 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.477700 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:52.477722 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.482888 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40673
I20260928 23:15:52.482952 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40673 every 8 connection(s)
I20260928 23:15:52.483225 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:52.487845 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.483889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45127 (local address 127.0.98.62:33199)
0928 23:15:52.483952 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.483953 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.484171 (+ 218us) server_negotiation.cc:407] Connection header received
0928 23:15:52.484671 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.484672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.484683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.484746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:52.485413 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.485894 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.486774 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.486943 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.486996 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.487651 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:52.487653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:52.487653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:52.487657 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:52.487671 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:52.487675 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:52.487710 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.487762 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.487785 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:15:52.488025 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.483850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33199 (local address 127.0.98.2:45127)
0928 23:15:52.484058 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.484097 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.484563 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.484811 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.484814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.484822 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:52.485337 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.485342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.485954 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.485954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.486719 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.486721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.487000 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.487000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.487102 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.487538 (+ 436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:52.487538 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:52.487565 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:52.487586 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:52.487712 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:52.487714 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:52.487717 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.487759 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.487832 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260928 23:15:52.488448 18274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33199
I20260928 23:15:52.488530 18274 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:52.488695 18274 heartbeater.cc:511] Master 127.0.98.62:33199 requested a full tablet report, sending...
I20260928 23:15:52.489024 17955 ts_manager.cc:194] Registered new tserver with Master: 6578651d0bb94d308827f17052d4709b (127.0.98.2:40673)
I20260928 23:15:52.489437 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45127
I20260928 23:15:52.492214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18146
I20260928 23:15:52.492314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:52.492508 392 external_mini_cluster.cc:1428] Setting key 303d0ed7d4a90b44e550b5faecc71411
I20260928 23:15:52.493323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--builtin_ntp_servers=127.0.98.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:52.571233 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:52.571419 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:52.571439 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:52.572577 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:52.572613 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:52.572636 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:52.574004 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:52.574213 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:52.574478 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:52.579187 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:52.579267 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:52.579511 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:52.579628 18277 server_base.cc:1034] running on GCE node
I20260928 23:15:52.579821 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:52.580008 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:52.581135 18277 hybrid_clock.cc:648] HybridClock initialized: now 1790637352581125 us; error 26 us; skew 500 ppm
I20260928 23:15:52.582237 18277 webserver.cc:460] Webserver started at http://127.0.98.3:45983/ using document root <none> and password file <none>
I20260928 23:15:52.582466 18277 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:52.582510 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:52.582630 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:52.583653 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "39b627c9f562474a84ff1b4eeeda5d5c"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "0a3dd8bea3a9169f41f7124df3cfd30d"
server_key_iv: "740466c7439826b217c7f07e171af616"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.583992 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "39b627c9f562474a84ff1b4eeeda5d5c"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "0a3dd8bea3a9169f41f7124df3cfd30d"
server_key_iv: "740466c7439826b217c7f07e171af616"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.585121 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:52.585762 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.585918 18277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.585986 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "39b627c9f562474a84ff1b4eeeda5d5c"
format_stamp: "Formatted at 2026-09-28 23:15:52 on dist-test-slave-6vgd"
server_key: "0a3dd8bea3a9169f41f7124df3cfd30d"
server_key_iv: "740466c7439826b217c7f07e171af616"
server_key_version: "encryptionkey@0"
I20260928 23:15:52.586050 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:52.603446 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:52.603714 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:52.603837 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:52.604041 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:52.604328 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:52.604367 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.604396 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:52.604420 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:52.609856 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33971
I20260928 23:15:52.609910 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33971 every 8 connection(s)
I20260928 23:15:52.610199 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:52.614691 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.610808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36303 (local address 127.0.98.62:33199)
0928 23:15:52.610878 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.610880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.611092 (+ 212us) server_negotiation.cc:407] Connection header received
0928 23:15:52.611551 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.611552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.611561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.611608 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:52.612198 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.612684 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.613628 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.613808 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.613848 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.614490 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:52.614492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:52.614492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:52.614496 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:52.614509 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:52.614513 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:52.614549 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.614607 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.614632 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:15:52.614869 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.610802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33199 (local address 127.0.98.3:36303)
0928 23:15:52.610982 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.611015 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.611479 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.611661 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.611665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.611673 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:52.612141 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.612145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.612744 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.612744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.613562 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.613565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.613860 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.613860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.613960 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.614378 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:52.614379 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:52.614406 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:52.614427 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:52.614557 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:52.614559 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:52.614561 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.614604 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.614675 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 23:15:52.615279 18405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33199
I20260928 23:15:52.615358 18405 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:52.615505 18405 heartbeater.cc:511] Master 127.0.98.62:33199 requested a full tablet report, sending...
I20260928 23:15:52.615795 17955 ts_manager.cc:194] Registered new tserver with Master: 39b627c9f562474a84ff1b4eeeda5d5c (127.0.98.3:33971)
I20260928 23:15:52.616117 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36303
I20260928 23:15:52.617561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18277
I20260928 23:15:52.617640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:52.617801 392 external_mini_cluster.cc:1428] Setting key 2017f29489833cb56bdd3867d9e5f927
I20260928 23:15:52.618525 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:52.622318 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.619755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.62:33199)
0928 23:15:52.619804 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.619806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.619892 (+ 86us) server_negotiation.cc:407] Connection header received
0928 23:15:52.619953 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.619953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.619964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.620013 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:52.620157 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.620609 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.621347 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.621543 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.621594 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.622106 (+ 512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:52.622107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:52.622107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:52.622111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:52.622124 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:52.622128 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:52.622177 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.622225 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.622247 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:15:52.623546 17955 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:52.624967 17955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:52.630532 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.626795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40673 (local address 127.0.0.1:49698)
0928 23:15:52.626964 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.627002 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.627060 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.627425 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.627426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.627428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.627487 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.627489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.628438 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.628439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.629510 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.629512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.630270 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.630344 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.630357 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.630417 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.630463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260928 23:15:52.630548 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.626958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.0.98.2:40673)
0928 23:15:52.627289 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.627292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.627300 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:52.627329 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.627331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.627342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.627422 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.627566 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.628369 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.629596 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.630200 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630257 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.630344 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.630431 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.630464 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":83,"threads_started":1}
I20260928 23:15:52.630899 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.626795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33945 (local address 127.0.0.1:53510)
0928 23:15:52.626964 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.627002 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.627060 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.627415 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.627418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.627419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.627478 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.627480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.628878 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.628878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.629930 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.629933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.630716 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.630768 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.630771 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.630804 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.630837 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260928 23:15:52.631057 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.627125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.0.98.1:33945)
0928 23:15:52.627276 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.627280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.627288 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:52.627317 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.627318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.627329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.627402 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.627535 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.628798 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630002 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.630648 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630830 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.630923 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.630947 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.630983 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260928 23:15:52.631385 18208 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb1511511242e1ac61e8645af705af (DEFAULT_TABLE table=test-table [id=4a8592ad67f247f59f10e31d2b55c56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:52.631594 18077 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb1511511242e1ac61e8645af705af (DEFAULT_TABLE table=test-table [id=4a8592ad67f247f59f10e31d2b55c56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:52.631701 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb1511511242e1ac61e8645af705af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:52.631891 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb1511511242e1ac61e8645af705af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:52.634184 18419 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Bootstrap starting.
I20260928 23:15:52.634186 18420 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: Bootstrap starting.
I20260928 23:15:52.634615 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.626800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33971 (local address 127.0.0.1:50196)
0928 23:15:52.627606 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.627615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.627664 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.627811 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.627812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.627814 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.627852 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.627854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.629020 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.629021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.629871 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.629874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.634346 (+ 4472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.634347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.634433 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.634438 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.634495 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.634538 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":60,"threads_started":1}
I20260928 23:15:52.634819 18420 tablet_bootstrap.cc:654] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:52.634887 18419 tablet_bootstrap.cc:654] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:52.635047 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.627011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.98.3:33971)
0928 23:15:52.627155 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.627160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.627663 (+ 503us) server_negotiation.cc:407] Connection header received
0928 23:15:52.627730 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.627732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.627747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.627813 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.627904 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.628955 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.629937 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.630799 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.630862 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.630931 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.634921 (+ 3990us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.634967 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260928 23:15:52.635100 18420 log.cc:824] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:52.635216 18419 log.cc:824] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:52.635753 18339 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb1511511242e1ac61e8645af705af (DEFAULT_TABLE table=test-table [id=4a8592ad67f247f59f10e31d2b55c56f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:52.635880 18419 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: No bootstrap required, opened a new log
I20260928 23:15:52.635941 18419 ts_tablet_manager.cc:1397] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:52.636078 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb1511511242e1ac61e8645af705af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:52.637501 18420 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: No bootstrap required, opened a new log
I20260928 23:15:52.637559 18420 ts_tablet_manager.cc:1397] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:15:52.638381 18419 raft_consensus.cc:348] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.638545 18419 raft_consensus.cc:374] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:52.638576 18419 raft_consensus.cc:729] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6578651d0bb94d308827f17052d4709b, State: Initialized, Role: FOLLOWER
I20260928 23:15:52.639055 18419 consensus_queue.cc:260] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.639290 18423 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Bootstrap starting.
I20260928 23:15:52.639438 18419 ts_tablet_manager.cc:1428] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:15:52.639454 18274 heartbeater.cc:503] Master 127.0.98.62:33199 was elected leader, sending a full tablet report...
I20260928 23:15:52.639460 18420 raft_consensus.cc:348] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.639603 18420 raft_consensus.cc:374] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:52.639637 18420 raft_consensus.cc:729] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc55b9cc715a470fab493e8ca5b3cff8, State: Initialized, Role: FOLLOWER
I20260928 23:15:52.639751 18420 consensus_queue.cc:260] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.639987 18423 tablet_bootstrap.cc:654] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:52.640007 18420 ts_tablet_manager.cc:1428] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:52.640049 18143 heartbeater.cc:503] Master 127.0.98.62:33199 was elected leader, sending a full tablet report...
I20260928 23:15:52.640357 18423 log.cc:824] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:52.641036 18423 tablet_bootstrap.cc:492] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: No bootstrap required, opened a new log
I20260928 23:15:52.641086 18423 ts_tablet_manager.cc:1397] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:52.642808 18423 raft_consensus.cc:348] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.642937 18423 raft_consensus.cc:374] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:52.643013 18423 raft_consensus.cc:729] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39b627c9f562474a84ff1b4eeeda5d5c, State: Initialized, Role: FOLLOWER
I20260928 23:15:52.643133 18423 consensus_queue.cc:260] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.643368 18423 ts_tablet_manager.cc:1428] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:52.643414 18405 heartbeater.cc:503] Master 127.0.98.62:33199 was elected leader, sending a full tablet report...
W20260928 23:15:52.733749 18275 tablet.cc:2369] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:52.734653 18427 raft_consensus.cc:482] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:52.734767 18427 raft_consensus.cc:504] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.735144 18427 leader_election.cc:288] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc55b9cc715a470fab493e8ca5b3cff8 (127.0.98.1:33945), 6578651d0bb94d308827f17052d4709b (127.0.98.2:40673)
I20260928 23:15:52.737711 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.735300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33945 (local address 127.0.98.3:44531)
0928 23:15:52.735369 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.735382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.735428 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.735554 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.735555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.735556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.735623 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.735624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.736207 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.736207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.736944 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.736946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.737489 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.737557 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.737560 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.737603 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.737644 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:15:52.737711 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.735393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44531 (local address 127.0.98.1:33945)
0928 23:15:52.735449 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.735451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.735461 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:52.735488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.735489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.735501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.735547 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.735683 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.736142 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737027 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.737424 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737483 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.737492 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.737615 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.737646 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:15:52.737838 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.735325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40673 (local address 127.0.98.3:53995)
0928 23:15:52.735511 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.735520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:52.735582 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:52.735702 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:52.735703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:52.735704 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:52.735769 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.735770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.736351 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.736352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.737066 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:52.737068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.737604 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:52.737671 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.737673 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:52.737717 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:15:52.737750 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":110,"threads_started":1}
I20260928 23:15:52.737847 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.735465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53995 (local address 127.0.98.2:40673)
0928 23:15:52.735513 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.735515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.735581 (+ 66us) server_negotiation.cc:407] Connection header received
0928 23:15:52.735636 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.735637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.735645 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.735692 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:52.735834 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.736273 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737145 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.737535 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.737595 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.737604 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.737727 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.737762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:15:52.738080 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb1511511242e1ac61e8645af705af" candidate_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" is_pre_election: true
I20260928 23:15:52.738179 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb1511511242e1ac61e8645af705af" candidate_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6578651d0bb94d308827f17052d4709b" is_pre_election: true
I20260928 23:15:52.738231 18097 raft_consensus.cc:2454] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b627c9f562474a84ff1b4eeeda5d5c in term 0.
I20260928 23:15:52.738317 18228 raft_consensus.cc:2454] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39b627c9f562474a84ff1b4eeeda5d5c in term 0.
I20260928 23:15:52.738478 18295 leader_election.cc:302] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b627c9f562474a84ff1b4eeeda5d5c, cc55b9cc715a470fab493e8ca5b3cff8; no voters:
I20260928 23:15:52.738617 18427 raft_consensus.cc:2781] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:52.738675 18427 raft_consensus.cc:482] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:52.738694 18427 raft_consensus.cc:3037] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.739478 18427 raft_consensus.cc:504] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.739593 18427 leader_election.cc:288] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [CANDIDATE]: Term 1 election: Requested vote from peers cc55b9cc715a470fab493e8ca5b3cff8 (127.0.98.1:33945), 6578651d0bb94d308827f17052d4709b (127.0.98.2:40673)
I20260928 23:15:52.739743 18228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb1511511242e1ac61e8645af705af" candidate_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6578651d0bb94d308827f17052d4709b"
I20260928 23:15:52.739828 18228 raft_consensus.cc:3037] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.739918 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb1511511242e1ac61e8645af705af" candidate_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc55b9cc715a470fab493e8ca5b3cff8"
I20260928 23:15:52.739995 18097 raft_consensus.cc:3037] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:52.740584 18228 raft_consensus.cc:2454] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b627c9f562474a84ff1b4eeeda5d5c in term 1.
I20260928 23:15:52.740674 18097 raft_consensus.cc:2454] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39b627c9f562474a84ff1b4eeeda5d5c in term 1.
I20260928 23:15:52.740741 18293 leader_election.cc:302] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39b627c9f562474a84ff1b4eeeda5d5c, 6578651d0bb94d308827f17052d4709b; no voters:
I20260928 23:15:52.740818 18427 raft_consensus.cc:2781] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:52.740949 18427 raft_consensus.cc:686] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 1 LEADER]: Becoming Leader. State: Replica: 39b627c9f562474a84ff1b4eeeda5d5c, State: Running, Role: LEADER
I20260928 23:15:52.741031 18427 consensus_queue.cc:237] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } }
I20260928 23:15:52.741863 17955 catalog_manager.cc:5495] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 39b627c9f562474a84ff1b4eeeda5d5c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39b627c9f562474a84ff1b4eeeda5d5c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33971 } health_report { overall_health: HEALTHY } } }
I20260928 23:15:52.775193 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.772900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.98.3:33971)
0928 23:15:52.772961 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:52.772963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:52.772992 (+ 29us) server_negotiation.cc:407] Connection header received
0928 23:15:52.773035 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:52.773036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:52.773060 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:52.773122 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:52.773275 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.773755 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.774483 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:52.774662 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:52.774707 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:52.774871 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:52.774977 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:52.775037 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:52.775071 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:15:52.775089 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:15:52.777046 18228 raft_consensus.cc:1261] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 1 FOLLOWER]: Refusing update from remote peer 39b627c9f562474a84ff1b4eeeda5d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:52.777046 18097 raft_consensus.cc:1261] T ccdb1511511242e1ac61e8645af705af P cc55b9cc715a470fab493e8ca5b3cff8 [term 1 FOLLOWER]: Refusing update from remote peer 39b627c9f562474a84ff1b4eeeda5d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:52.777472 18429 consensus_queue.cc:1035] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6578651d0bb94d308827f17052d4709b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:52.777611 18427 consensus_queue.cc:1035] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc55b9cc715a470fab493e8ca5b3cff8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:52.780066 18433 mvcc.cc:205] Tried to move back new op lower bound from 7334450596972601344 to 7334450596828327936. Current Snapshot: MvccSnapshot[applied={T|T < 7334450596972601344}]
I20260928 23:15:52.780834 18434 mvcc.cc:205] Tried to move back new op lower bound from 7334450596972601344 to 7334450596828327936. Current Snapshot: MvccSnapshot[applied={T|T < 7334450596972601344}]
I20260928 23:15:52.782491 17954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:52.782604 17954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:52.783375 17954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2fbc6b6c3404b5bb1265ff8137d5272: Sending DeleteTablet for 3 replicas of tablet ccdb1511511242e1ac61e8645af705af
I20260928 23:15:52.783723 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdb1511511242e1ac61e8645af705af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='slave'} at 127.0.0.1:49698
I20260928 23:15:52.783721 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdb1511511242e1ac61e8645af705af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='slave'} at 127.0.0.1:53510
I20260928 23:15:52.783777 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdb1511511242e1ac61e8645af705af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:52 UTC) from {username='slave'} at 127.0.0.1:50196
I20260928 23:15:52.783964 18444 tablet_replica.cc:330] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: stopping tablet replica
I20260928 23:15:52.784065 18444 raft_consensus.cc:2229] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:52.784169 18444 raft_consensus.cc:2258] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:52.784183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18015
I20260928 23:15:52.784613 18444 ts_tablet_manager.cc:1905] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:52.786422 18444 ts_tablet_manager.cc:1918] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:52.786495 18444 log.cc:1196] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/ccdb1511511242e1ac61e8645af705af
I20260928 23:15:52.786748 18444 ts_tablet_manager.cc:1939] T ccdb1511511242e1ac61e8645af705af P 39b627c9f562474a84ff1b4eeeda5d5c: Deleting consensus metadata
I20260928 23:15:52.787113 17941 catalog_manager.cc:4841] TS 39b627c9f562474a84ff1b4eeeda5d5c (127.0.98.3:33971): tablet ccdb1511511242e1ac61e8645af705af (table test-table [id=4a8592ad67f247f59f10e31d2b55c56f]) successfully deleted
I20260928 23:15:52.789361 18442 tablet_replica.cc:330] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: stopping tablet replica
I20260928 23:15:52.789484 18442 raft_consensus.cc:2229] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:52.789597 18442 raft_consensus.cc:2258] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 23:15:52.790035 17942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33945 (error 108)
I20260928 23:15:52.790128 18442 ts_tablet_manager.cc:1905] T ccdb1511511242e1ac61e8645af705af P 6578651d0bb94d308827f17052d4709b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:52.790237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18146
I20260928 23:15:52.793814 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.793672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33945 (local address 127.0.0.1:53518)
0928 23:15:52.793738 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.793782 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:15:52.793964 17942 catalog_manager.cc:4584] TS cc55b9cc715a470fab493e8ca5b3cff8 (127.0.98.1:33945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccdb1511511242e1ac61e8645af705af: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33945: connect: Connection refused (error 111)
I20260928 23:15:52.794961 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:52.794874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40673 (local address 127.0.0.1:49714)
0928 23:15:52.794913 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:52.794938 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:15:52.795264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18277
I20260928 23:15:52.799383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 17924
2026-09-28T23:15:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (709 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:15:52 dist-test-slave-6vgd krb5kdc[18448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:15:52 dist-test-slave-6vgd krb5kdc[18448](info): set up 2 sockets
Sep 28 23:15:52 dist-test-slave-6vgd krb5kdc[18448](info): commencing operation
krb5kdc: starting...
W20260928 23:15:53.744630 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:15:53 dist-test-slave-6vgd krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:15:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:15:53.855819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:53.933881 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:53.934104 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:53.934136 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:53.935297 18461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:53.935340 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:53.935359 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:53.935374 18461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:53.935389 18461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:53.936872 18461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:53.937120 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:53.937371 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:53.941811 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:53.941830 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:53.941812 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:53.942044 18461 server_base.cc:1034] running on GCE node
I20260928 23:15:53.942305 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:53.942566 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:53.943708 18461 hybrid_clock.cc:648] HybridClock initialized: now 1790637353943697 us; error 28 us; skew 500 ppm
Sep 28 23:15:53 dist-test-slave-6vgd krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:53.951164 18461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:15:53.951591 18461 webserver.cc:460] Webserver started at http://127.0.98.62:40383/ using document root <none> and password file <none>
I20260928 23:15:53.951793 18461 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:53.951838 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:53.951944 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:53.953009 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "15735b21bfff4ea6909b05e9d51607a1"
format_stamp: "Formatted at 2026-09-28 23:15:53 on dist-test-slave-6vgd"
server_key: "5624c5fb11f2028a3398f39e2d9cfa9e"
server_key_iv: "8077662fc284ba2ab6f6610bbbafb87e"
server_key_version: "encryptionkey@0"
I20260928 23:15:53.953367 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "15735b21bfff4ea6909b05e9d51607a1"
format_stamp: "Formatted at 2026-09-28 23:15:53 on dist-test-slave-6vgd"
server_key: "5624c5fb11f2028a3398f39e2d9cfa9e"
server_key_iv: "8077662fc284ba2ab6f6610bbbafb87e"
server_key_version: "encryptionkey@0"
I20260928 23:15:53.954633 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 23:15:53.955307 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:53.955453 18461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:15:53.955529 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "15735b21bfff4ea6909b05e9d51607a1"
format_stamp: "Formatted at 2026-09-28 23:15:53 on dist-test-slave-6vgd"
server_key: "5624c5fb11f2028a3398f39e2d9cfa9e"
server_key_iv: "8077662fc284ba2ab6f6610bbbafb87e"
server_key_version: "encryptionkey@0"
I20260928 23:15:53.955591 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:53.963562 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:53.965106 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:53.965260 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:53.968853 18461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32787
I20260928 23:15:53.968909 18528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32787 every 8 connection(s)
I20260928 23:15:53.969239 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:53.970075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18461
I20260928 23:15:53.970150 18529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:53.970201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:53.970417 392 external_mini_cluster.cc:1428] Setting key 7c0eefd13bd828a019b2d9b407b6d0b4
I20260928 23:15:53.973325 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: Bootstrap starting.
I20260928 23:15:53.973959 18529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:53.974252 18529 log.cc:824] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:53.974918 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: No bootstrap required, opened a new log
Sep 28 23:15:53 dist-test-slave-6vgd krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:53.977198 18529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } }
I20260928 23:15:53.977415 18529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:53.977463 18529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15735b21bfff4ea6909b05e9d51607a1, State: Initialized, Role: FOLLOWER
I20260928 23:15:53.977594 18529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } }
I20260928 23:15:53.977685 18529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:53.977726 18529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:53.977774 18529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:53.978386 18529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } }
I20260928 23:15:53.978483 18529 leader_election.cc:302] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15735b21bfff4ea6909b05e9d51607a1; no voters:
I20260928 23:15:53.978636 18529 leader_election.cc:288] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:53.978682 18534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:53.978824 18534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [term 1 LEADER]: Becoming Leader. State: Replica: 15735b21bfff4ea6909b05e9d51607a1, State: Running, Role: LEADER
I20260928 23:15:53.978920 18529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:53.978967 18534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } }
I20260928 23:15:53.979369 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15735b21bfff4ea6909b05e9d51607a1. Latest consensus state: current_term: 1 leader_uuid: "15735b21bfff4ea6909b05e9d51607a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } } }
I20260928 23:15:53.979418 18536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15735b21bfff4ea6909b05e9d51607a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15735b21bfff4ea6909b05e9d51607a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32787 } } }
I20260928 23:15:53.979525 18536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:53.979739 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:53.980252 18543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:53.980727 18543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:53.981492 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:53.970938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.62:32787)
0928 23:15:53.971117 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:53.971125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:53.971150 (+ 25us) server_negotiation.cc:407] Connection header received
0928 23:15:53.971593 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:53.971599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:53.971655 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:53.971771 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:53.972585 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:53.973600 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:53.974597 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:53.975070 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:53.975164 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:53.977707 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:53.977711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:53.977711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:53.977723 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:53.979677 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:53.980041 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:53.980044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:53.980044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:53.980145 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:53.980367 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:53.980368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:53.980369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:53.980641 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:53.980782 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:53.981135 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 23:15:53.981293 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260928 23:15:53.982465 18543 catalog_manager.cc:1347] Generated new cluster ID: f6557eef59674e8c89ea5e20b1975187
I20260928 23:15:53.982514 18543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:53.993608 18543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:53.994163 18543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:53.999222 18543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: Generated new TSK 0
I20260928 23:15:53.999393 18543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:15:54.027493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:54.105984 18555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.106187 18555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.106230 18555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.107385 18555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.107425 18555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.107460 18555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:54.108934 18555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.109162 18555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.109398 18555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.114008 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.113979 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.114082 18555 server_base.cc:1034] running on GCE node
W20260928 23:15:54.113979 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.114410 18555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.114626 18555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.115760 18555 hybrid_clock.cc:648] HybridClock initialized: now 1790637354115754 us; error 35 us; skew 500 ppm
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:54.118285 18555 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:15:54.118707 18555 webserver.cc:460] Webserver started at http://127.0.98.1:39841/ using document root <none> and password file <none>
I20260928 23:15:54.118914 18555 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.118961 18555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.119099 18555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.120124 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9430e7829674119a13cacb258ff0cf1"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6249991067ce4d9ab3ed520a904ad228"
server_key_iv: "4146bbdf72888dda2031130b0f07aa3a"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.120453 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9430e7829674119a13cacb258ff0cf1"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6249991067ce4d9ab3ed520a904ad228"
server_key_iv: "4146bbdf72888dda2031130b0f07aa3a"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.121702 18555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:54.122429 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.122575 18555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:54.122649 18555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "d9430e7829674119a13cacb258ff0cf1"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6249991067ce4d9ab3ed520a904ad228"
server_key_iv: "4146bbdf72888dda2031130b0f07aa3a"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.122711 18555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:54.160810 18555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:54.162672 18555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:54.162810 18555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:54.163023 18555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:54.163308 18555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:54.163341 18555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.163369 18555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:54.163391 18555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.168887 18555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39297
I20260928 23:15:54.168953 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39297 every 8 connection(s)
I20260928 23:15:54.169275 18555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:54.172473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18555
I20260928 23:15:54.172590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:54.172772 392 external_mini_cluster.cc:1428] Setting key 4863b33a4de467b099c77820ba60f802
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:54.178714 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.170089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57331 (local address 127.0.98.62:32787)
0928 23:15:54.170175 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.170178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.170371 (+ 193us) server_negotiation.cc:407] Connection header received
0928 23:15:54.171336 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.171338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.171350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.171402 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.172043 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.172548 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.173860 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.174076 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.174131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.176131 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.176133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.176133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:54.176141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.177472 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.177822 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.177823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.177824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.177874 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.178081 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.178082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.178083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.178189 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.178250 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.178511 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.178620 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 23:15:54.178845 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.170051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32787 (local address 127.0.98.1:57331)
0928 23:15:54.170249 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.170288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.171222 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.171483 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.171487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.171750 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:54.171947 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.171952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.172632 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.172633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.173782 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.173785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.174145 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.174146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.174275 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.174747 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:54.174755 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:54.176043 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:54.177543 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.177551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.177553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.177759 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.177925 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.177926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.177926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.178028 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.178248 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:54.178255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:54.178388 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.178482 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.178630 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260928 23:15:54.179536 18684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32787
I20260928 23:15:54.179620 18684 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:54.179808 18684 heartbeater.cc:511] Master 127.0.98.62:32787 requested a full tablet report, sending...
I20260928 23:15:54.180389 18493 ts_manager.cc:194] Registered new tserver with Master: d9430e7829674119a13cacb258ff0cf1 (127.0.98.1:39297)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 23:15:54.181162 18493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:15:54.196115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:54.274850 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.275059 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.275080 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.276198 18689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.276237 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.276265 18689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:54.277722 18689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.277948 18689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.278156 18689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.282894 18695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.282891 18694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.282892 18697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.283179 18689 server_base.cc:1034] running on GCE node
I20260928 23:15:54.283373 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.283576 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.284708 18689 hybrid_clock.cc:648] HybridClock initialized: now 1790637354284695 us; error 31 us; skew 500 ppm
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:54.287068 18689 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:15:54.287493 18689 webserver.cc:460] Webserver started at http://127.0.98.2:34161/ using document root <none> and password file <none>
I20260928 23:15:54.287771 18689 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.287822 18689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.287914 18689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.288921 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "ddf609143fcc4520911119192261fd03"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "5660248818dfb7ba2d47e7db120030ec"
server_key_iv: "8a90bdc4f1f7d9f363d69fc70288ecdc"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.289268 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ddf609143fcc4520911119192261fd03"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "5660248818dfb7ba2d47e7db120030ec"
server_key_iv: "8a90bdc4f1f7d9f363d69fc70288ecdc"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.290454 18689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:15:54.291090 18704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.291255 18689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:54.291333 18689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "ddf609143fcc4520911119192261fd03"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "5660248818dfb7ba2d47e7db120030ec"
server_key_iv: "8a90bdc4f1f7d9f363d69fc70288ecdc"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.291384 18689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:54.305061 18689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:54.307121 18689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:54.307230 18689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:54.307399 18689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:54.307651 18689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:54.307680 18689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.307701 18689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:54.307714 18689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.312857 18689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36311
I20260928 23:15:54.312952 18817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36311 every 8 connection(s)
I20260928 23:15:54.313179 18689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:54.320609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18689
I20260928 23:15:54.320732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:54.320915 392 external_mini_cluster.cc:1428] Setting key 7c4a0ea232f59d90076dcdf1382a1ac6
I20260928 23:15:54.321925 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.313828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32965 (local address 127.0.98.62:32787)
0928 23:15:54.313926 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.313929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.314069 (+ 140us) server_negotiation.cc:407] Connection header received
0928 23:15:54.314927 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.314928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.314938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.314988 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.315523 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.316070 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.317096 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.317346 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.317406 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.319366 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.319368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.319368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:54.319374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.320545 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.320991 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.320992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.320992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.321047 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.321295 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.321296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.321296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.321413 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.321475 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.321772 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.321846 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:15:54.322081 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.313794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32787 (local address 127.0.98.2:32965)
0928 23:15:54.313955 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.313995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.314839 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.315043 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.315046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.315275 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:54.315462 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.315467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.316136 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.316137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.316996 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.316999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.317409 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.317409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.317507 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.317969 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:54.317979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:54.319285 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:54.320626 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.320633 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.320635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.320895 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.321102 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.321102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.321103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.321247 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.321469 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:54.321476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:54.321642 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.321731 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.321856 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 23:15:54.322834 18818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32787
I20260928 23:15:54.322947 18818 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:54.323135 18818 heartbeater.cc:511] Master 127.0.98.62:32787 requested a full tablet report, sending...
I20260928 23:15:54.323519 18493 ts_manager.cc:194] Registered new tserver with Master: ddf609143fcc4520911119192261fd03 (127.0.98.2:36311)
I20260928 23:15:54.323868 18493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32965
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:15:54.343350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:54.420986 18823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.421180 18823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.421231 18823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.422466 18823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.422528 18823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.422590 18823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:54.424259 18823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.424487 18823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.424688 18823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.429466 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.429529 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.429486 18828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.429765 18823 server_base.cc:1034] running on GCE node
I20260928 23:15:54.429960 18823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.430171 18823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.431305 18823 hybrid_clock.cc:648] HybridClock initialized: now 1790637354431292 us; error 23 us; skew 500 ppm
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:54.433686 18823 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:15:54.434103 18823 webserver.cc:460] Webserver started at http://127.0.98.3:40865/ using document root <none> and password file <none>
I20260928 23:15:54.434293 18823 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.434338 18823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.434466 18823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.435453 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ae91de837664bb0885e2a2a95ef2f10"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "2c7be2ae013eac73a172dbf42509c364"
server_key_iv: "3da548b0629d1b1839451f0ef27ef328"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.435768 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae91de837664bb0885e2a2a95ef2f10"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "2c7be2ae013eac73a172dbf42509c364"
server_key_iv: "3da548b0629d1b1839451f0ef27ef328"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.436936 18823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:15:54.437606 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.437748 18823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.437825 18823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "8ae91de837664bb0885e2a2a95ef2f10"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "2c7be2ae013eac73a172dbf42509c364"
server_key_iv: "3da548b0629d1b1839451f0ef27ef328"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.437889 18823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:54.475134 18823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:54.477285 18823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:54.477432 18823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:54.477648 18823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:54.477941 18823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:54.477974 18823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.478001 18823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:54.478019 18823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.483755 18823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38991
I20260928 23:15:54.483795 18951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38991 every 8 connection(s)
I20260928 23:15:54.484086 18823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:54.487649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18823
I20260928 23:15:54.487751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:54.487916 392 external_mini_cluster.cc:1428] Setting key 0651c8842b1486598b58f1de0f23e94e
Sep 28 23:15:54 dist-test-slave-6vgd krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:54.492981 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.484718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59827 (local address 127.0.98.62:32787)
0928 23:15:54.484790 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.484792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.484952 (+ 160us) server_negotiation.cc:407] Connection header received
0928 23:15:54.485876 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.485877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.485890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.485950 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.486759 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.487267 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.488620 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.488828 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.488878 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.490799 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.490801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.490802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:54.490809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.491788 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.492140 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.492141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.492142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.492193 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.492397 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.492398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.492399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.492505 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.492561 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.492806 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.492883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:15:54.493127 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.484698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32787 (local address 127.0.98.3:59827)
0928 23:15:54.484836 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.484871 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.485788 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.486068 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.486073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.486413 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:54.486682 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.486688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.487345 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.487346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.488555 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.488558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.488901 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.488902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.489028 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.489466 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:54.489474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:54.490717 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:54.491857 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.491862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.491864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.492074 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.492245 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:54.492246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:54.492246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:54.492345 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:54.492558 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:54.492564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:54.492695 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.492779 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.492924 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260928 23:15:54.493527 18952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32787
I20260928 23:15:54.493610 18952 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:54.493772 18952 heartbeater.cc:511] Master 127.0.98.62:32787 requested a full tablet report, sending...
I20260928 23:15:54.494095 18493 ts_manager.cc:194] Registered new tserver with Master: 8ae91de837664bb0885e2a2a95ef2f10 (127.0.98.3:38991)
I20260928 23:15:54.494463 18493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59827
I20260928 23:15:54.499455 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:54.506029 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.500579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.0.98.62:32787)
0928 23:15:54.500657 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.500660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.500749 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:15:54.500799 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.500800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.500809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.500854 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.501410 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.501886 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.502576 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.502767 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.502818 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.503950 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.503951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.503952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:54.503958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.504890 (+ 932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.505282 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.505284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.505284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.505334 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:54.505522 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:54.505523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:54.505523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:54.505652 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.505743 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.505868 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.505936 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:15:54.507447 18493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:54.508919 18493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:54.515033 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.511072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.0.98.2:36311)
0928 23:15:54.511256 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.511262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.511271 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:54.511313 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.511315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.511329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.511397 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:54.511641 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.512671 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514178 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.514657 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514719 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.514777 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.514904 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.514955 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
I20260928 23:15:54.515036 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.510961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36311 (local address 127.0.0.1:33596)
0928 23:15:54.511086 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.511125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.511193 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.511501 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.511504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.511507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:54.511577 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.511578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.512767 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.512768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.514074 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.514077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.514732 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.514809 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.514821 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.514884 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.514939 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":6,"thread_start_us":62,"threads_started":1}
I20260928 23:15:54.515196 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.511246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60598 (local address 127.0.98.3:38991)
0928 23:15:54.511636 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.511640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.511650 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:54.511682 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.511684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.511697 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.511767 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:54.512013 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.512958 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514016 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.514676 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514740 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.514794 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.515071 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.515112 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":72,"threads_started":1}
I20260928 23:15:54.515225 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.511145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38991 (local address 127.0.0.1:60598)
0928 23:15:54.511395 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.511405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.511445 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.511850 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.511851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.511853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:54.511929 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.511931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.513041 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.513042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.513935 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.513938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.514828 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.514888 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.514892 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.514936 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.514976 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260928 23:15:54.515637 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.511290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39297 (local address 127.0.0.1:52880)
0928 23:15:54.511862 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.511872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.511916 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.512095 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.512096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.512098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:54.512136 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.512137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.512996 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.512997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.514119 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.514121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.515024 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.515025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.515193 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.515198 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.515532 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.515575 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":23,"thread_start_us":57,"threads_started":1}
I20260928 23:15:54.515689 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.511414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52880 (local address 127.0.98.1:39297)
0928 23:15:54.511590 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.511593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.511968 (+ 375us) server_negotiation.cc:407] Connection header received
0928 23:15:54.512012 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.512014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.512029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.512099 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:54.512179 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.512892 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514190 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.514885 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.514944 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.514993 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.515568 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.515617 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260928 23:15:54.515930 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 (DEFAULT_TABLE table=test-table [id=86cdf0ab90d44e5a8f06c120bc81668b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:54.515949 18752 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 (DEFAULT_TABLE table=test-table [id=86cdf0ab90d44e5a8f06c120bc81668b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:54.516251 18752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5a4e5f1b874cd497fa725e54667ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:54.516255 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5a4e5f1b874cd497fa725e54667ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:54.516379 18618 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 (DEFAULT_TABLE table=test-table [id=86cdf0ab90d44e5a8f06c120bc81668b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:54.516655 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5a4e5f1b874cd497fa725e54667ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:54.518695 18969 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Bootstrap starting.
I20260928 23:15:54.519349 18969 tablet_bootstrap.cc:654] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:54.519366 18970 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Bootstrap starting.
I20260928 23:15:54.519605 18971 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: Bootstrap starting.
I20260928 23:15:54.519984 18970 tablet_bootstrap.cc:654] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:54.520083 18969 log.cc:824] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:54.520190 18971 tablet_bootstrap.cc:654] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:54.520478 18970 log.cc:824] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:54.520689 18971 log.cc:824] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:54.520817 18969 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: No bootstrap required, opened a new log
I20260928 23:15:54.520876 18969 ts_tablet_manager.cc:1397] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:54.521247 18970 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: No bootstrap required, opened a new log
I20260928 23:15:54.521340 18970 ts_tablet_manager.cc:1397] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:54.521340 18971 tablet_bootstrap.cc:492] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: No bootstrap required, opened a new log
I20260928 23:15:54.521400 18971 ts_tablet_manager.cc:1397] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:54.522776 18971 raft_consensus.cc:348] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.522806 18969 raft_consensus.cc:348] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.522933 18971 raft_consensus.cc:374] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:54.522965 18971 raft_consensus.cc:729] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9430e7829674119a13cacb258ff0cf1, State: Initialized, Role: FOLLOWER
I20260928 23:15:54.522961 18969 raft_consensus.cc:374] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:54.523005 18969 raft_consensus.cc:729] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf609143fcc4520911119192261fd03, State: Initialized, Role: FOLLOWER
I20260928 23:15:54.523094 18971 consensus_queue.cc:260] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.523113 18969 consensus_queue.cc:260] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.523335 18971 ts_tablet_manager.cc:1428] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:54.523355 18969 ts_tablet_manager.cc:1428] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 23:15:54.523439 18684 heartbeater.cc:503] Master 127.0.98.62:32787 was elected leader, sending a full tablet report...
I20260928 23:15:54.523465 18818 heartbeater.cc:503] Master 127.0.98.62:32787 was elected leader, sending a full tablet report...
I20260928 23:15:54.523583 18970 raft_consensus.cc:348] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.523737 18970 raft_consensus.cc:374] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:54.523772 18970 raft_consensus.cc:729] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae91de837664bb0885e2a2a95ef2f10, State: Initialized, Role: FOLLOWER
I20260928 23:15:54.523882 18970 consensus_queue.cc:260] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.524097 18970 ts_tablet_manager.cc:1428] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 23:15:54.524128 18952 heartbeater.cc:503] Master 127.0.98.62:32787 was elected leader, sending a full tablet report...
I20260928 23:15:54.552404 18977 raft_consensus.cc:482] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:54.552531 18977 raft_consensus.cc:504] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.552834 18977 leader_election.cc:288] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddf609143fcc4520911119192261fd03 (127.0.98.2:36311), d9430e7829674119a13cacb258ff0cf1 (127.0.98.1:39297)
I20260928 23:15:54.556692 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.552997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36311 (local address 127.0.98.3:38341)
0928 23:15:54.553065 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.553077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.553130 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.553357 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.553358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.553359 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:54.553430 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.553432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.554341 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.554341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.555676 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.555679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.556399 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.556399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.556490 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.556494 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.556546 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.556597 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":102}
I20260928 23:15:54.556706 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.553152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38341 (local address 127.0.98.2:36311)
0928 23:15:54.553240 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.553243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.553250 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:15:54.553277 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.553278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.553290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.553344 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:54.553497 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.554259 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.555767 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.556323 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.556394 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.556405 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.556587 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.556635 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 23:15:54.556706 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.553169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47121 (local address 127.0.98.1:39297)
0928 23:15:54.553236 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.553237 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.553334 (+ 97us) server_negotiation.cc:407] Connection header received
0928 23:15:54.553399 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.553400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.553411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.553448 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:54.553577 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.554258 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.555767 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.556330 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.556394 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.556405 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.556587 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.556635 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:15:54.556702 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.553090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39297 (local address 127.0.98.3:47121)
0928 23:15:54.553264 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.553273 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.553327 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.553463 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.553464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.553465 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:54.553519 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.553520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.554338 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.554339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.555676 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.555679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.556399 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.556399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.556490 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.556494 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.556546 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.556597 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":53,"thread_start_us":129,"threads_started":1}
I20260928 23:15:54.557077 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5a4e5f1b874cd497fa725e54667ef9" candidate_uuid: "8ae91de837664bb0885e2a2a95ef2f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf609143fcc4520911119192261fd03" is_pre_election: true
I20260928 23:15:54.557251 18772 raft_consensus.cc:2454] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae91de837664bb0885e2a2a95ef2f10 in term 0.
I20260928 23:15:54.557299 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5a4e5f1b874cd497fa725e54667ef9" candidate_uuid: "8ae91de837664bb0885e2a2a95ef2f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9430e7829674119a13cacb258ff0cf1" is_pre_election: true
I20260928 23:15:54.557443 18638 raft_consensus.cc:2454] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae91de837664bb0885e2a2a95ef2f10 in term 0.
I20260928 23:15:54.557512 18839 leader_election.cc:302] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae91de837664bb0885e2a2a95ef2f10, ddf609143fcc4520911119192261fd03; no voters:
I20260928 23:15:54.557629 18977 raft_consensus.cc:2781] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:54.557689 18977 raft_consensus.cc:482] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:54.557713 18977 raft_consensus.cc:3037] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:54.558389 18977 raft_consensus.cc:504] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.558499 18977 leader_election.cc:288] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [CANDIDATE]: Term 1 election: Requested vote from peers ddf609143fcc4520911119192261fd03 (127.0.98.2:36311), d9430e7829674119a13cacb258ff0cf1 (127.0.98.1:39297)
I20260928 23:15:54.558673 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5a4e5f1b874cd497fa725e54667ef9" candidate_uuid: "8ae91de837664bb0885e2a2a95ef2f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9430e7829674119a13cacb258ff0cf1"
I20260928 23:15:54.558668 18772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5a4e5f1b874cd497fa725e54667ef9" candidate_uuid: "8ae91de837664bb0885e2a2a95ef2f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddf609143fcc4520911119192261fd03"
I20260928 23:15:54.558748 18772 raft_consensus.cc:3037] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:54.558748 18638 raft_consensus.cc:3037] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:54.559649 18638 raft_consensus.cc:2454] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae91de837664bb0885e2a2a95ef2f10 in term 1.
I20260928 23:15:54.559649 18772 raft_consensus.cc:2454] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae91de837664bb0885e2a2a95ef2f10 in term 1.
I20260928 23:15:54.559778 18839 leader_election.cc:302] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae91de837664bb0885e2a2a95ef2f10, d9430e7829674119a13cacb258ff0cf1; no voters:
I20260928 23:15:54.559959 18977 raft_consensus.cc:2781] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:54.560142 18977 raft_consensus.cc:686] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 1 LEADER]: Becoming Leader. State: Replica: 8ae91de837664bb0885e2a2a95ef2f10, State: Running, Role: LEADER
I20260928 23:15:54.560230 18977 consensus_queue.cc:237] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } }
I20260928 23:15:54.561008 18491 catalog_manager.cc:5495] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae91de837664bb0885e2a2a95ef2f10 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8ae91de837664bb0885e2a2a95ef2f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae91de837664bb0885e2a2a95ef2f10" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 } health_report { overall_health: UNKNOWN } } }
W20260928 23:15:54.563673 18819 tablet.cc:2369] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:54.575690 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.573483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.0.98.3:38991)
0928 23:15:54.573529 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.573531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.573540 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:54.573610 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.573611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.573629 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.573688 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:54.573804 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.574255 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.575000 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.575182 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.575234 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.575377 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:54.575463 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:54.575544 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.575590 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.575598 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:15:54.577384 18638 raft_consensus.cc:1261] T 3a5a4e5f1b874cd497fa725e54667ef9 P d9430e7829674119a13cacb258ff0cf1 [term 1 FOLLOWER]: Refusing update from remote peer 8ae91de837664bb0885e2a2a95ef2f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:54.577383 18772 raft_consensus.cc:1261] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 1 FOLLOWER]: Refusing update from remote peer 8ae91de837664bb0885e2a2a95ef2f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:54.577701 18977 consensus_queue.cc:1035] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddf609143fcc4520911119192261fd03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:54.577863 18979 consensus_queue.cc:1035] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9430e7829674119a13cacb258ff0cf1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:54.580518 18984 mvcc.cc:205] Tried to move back new op lower bound from 7334450604346761216 to 7334450604279607296. Current Snapshot: MvccSnapshot[applied={T|T < 7334450604346761216}]
I20260928 23:15:54.581152 18983 mvcc.cc:205] Tried to move back new op lower bound from 7334450604346761216 to 7334450604279607296. Current Snapshot: MvccSnapshot[applied={T|T < 7334450604346761216 or (T in {7334450604346761216})}]
I20260928 23:15:54.582353 18491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:54.582432 18491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:54.583117 18491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15735b21bfff4ea6909b05e9d51607a1: Sending DeleteTablet for 3 replicas of tablet 3a5a4e5f1b874cd497fa725e54667ef9
I20260928 23:15:54.583400 18752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33596
I20260928 23:15:54.583483 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60598
I20260928 23:15:54.583598 18991 tablet_replica.cc:330] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: stopping tablet replica
I20260928 23:15:54.583691 18991 raft_consensus.cc:2229] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:54.583693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18555
I20260928 23:15:54.583698 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5a4e5f1b874cd497fa725e54667ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52880
I20260928 23:15:54.583711 18992 tablet_replica.cc:330] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: stopping tablet replica
I20260928 23:15:54.583794 18991 raft_consensus.cc:2258] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:54.583839 18992 raft_consensus.cc:2229] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:54.583927 18992 raft_consensus.cc:2258] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:54.584816 18992 ts_tablet_manager.cc:1905] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:54.585839 18991 ts_tablet_manager.cc:1905] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:54.586529 18992 ts_tablet_manager.cc:1918] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:54.586623 18992 log.cc:1196] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/3a5a4e5f1b874cd497fa725e54667ef9
I20260928 23:15:54.586876 18992 ts_tablet_manager.cc:1939] T 3a5a4e5f1b874cd497fa725e54667ef9 P 8ae91de837664bb0885e2a2a95ef2f10: Deleting consensus metadata
I20260928 23:15:54.587075 18991 ts_tablet_manager.cc:1918] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:54.587131 18991 log.cc:1196] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/3a5a4e5f1b874cd497fa725e54667ef9
I20260928 23:15:54.587303 18477 catalog_manager.cc:4841] TS 8ae91de837664bb0885e2a2a95ef2f10 (127.0.98.3:38991): tablet 3a5a4e5f1b874cd497fa725e54667ef9 (table test-table [id=86cdf0ab90d44e5a8f06c120bc81668b]) successfully deleted
I20260928 23:15:54.587378 18991 ts_tablet_manager.cc:1939] T 3a5a4e5f1b874cd497fa725e54667ef9 P ddf609143fcc4520911119192261fd03: Deleting consensus metadata
I20260928 23:15:54.587692 18477 catalog_manager.cc:4841] TS ddf609143fcc4520911119192261fd03 (127.0.98.2:36311): tablet 3a5a4e5f1b874cd497fa725e54667ef9 (table test-table [id=86cdf0ab90d44e5a8f06c120bc81668b]) successfully deleted
W20260928 23:15:54.588483 18477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39297 (error 108)
I20260928 23:15:54.588864 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.588774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39297 (local address 127.0.0.1:52896)
0928 23:15:54.588807 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.588837 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 23:15:54.588972 18477 catalog_manager.cc:4584] TS d9430e7829674119a13cacb258ff0cf1 (127.0.98.1:39297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a5a4e5f1b874cd497fa725e54667ef9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39297: connect: Connection refused (error 111)
I20260928 23:15:54.589452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18689
I20260928 23:15:54.593715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18823
I20260928 23:15:54.598060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18461
2026-09-28T23:15:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1799 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T23:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:54Z Disabled control of system clock
I20260928 23:15:54.622758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:54.700165 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.700383 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.700403 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.701511 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:54.701545 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.701555 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.701566 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:54.701577 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:54.702832 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.703017 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.703248 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.707847 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.707928 18997 server_base.cc:1034] running on GCE node
W20260928 23:15:54.707845 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.707940 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.708300 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.708571 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.709725 18997 hybrid_clock.cc:648] HybridClock initialized: now 1790637354709680 us; error 41 us; skew 500 ppm
I20260928 23:15:54.710867 18997 webserver.cc:460] Webserver started at http://127.0.98.62:45685/ using document root <none> and password file <none>
I20260928 23:15:54.711081 18997 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.711128 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.711248 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.712246 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "4c9772300fce40469e9bb333b4fc0fdc"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6a3b755f4ae71d9a436fd6e8fa9302bc"
server_key_iv: "bd2fdfe35b15e5e3784ac4f04de48dce"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.712554 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c9772300fce40469e9bb333b4fc0fdc"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6a3b755f4ae71d9a436fd6e8fa9302bc"
server_key_iv: "bd2fdfe35b15e5e3784ac4f04de48dce"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.713740 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:54.714342 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.714478 18997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:54.714553 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "4c9772300fce40469e9bb333b4fc0fdc"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6a3b755f4ae71d9a436fd6e8fa9302bc"
server_key_iv: "bd2fdfe35b15e5e3784ac4f04de48dce"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.714609 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:54.722967 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:54.723250 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:54.723371 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:54.726845 18997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39185
I20260928 23:15:54.726874 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39185 every 8 connection(s)
I20260928 23:15:54.727150 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:54.728029 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:54.730166 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: Bootstrap starting.
I20260928 23:15:54.730675 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:54.730930 19064 log.cc:824] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:54.731498 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: No bootstrap required, opened a new log
I20260928 23:15:54.733019 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260928 23:15:54.733147 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:54.733170 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c9772300fce40469e9bb333b4fc0fdc, State: Initialized, Role: FOLLOWER
I20260928 23:15:54.733328 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260928 23:15:54.733403 19064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:54.733438 19064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:54.733482 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:54.733938 19064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260928 23:15:54.734018 19064 leader_election.cc:302] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c9772300fce40469e9bb333b4fc0fdc; no voters:
I20260928 23:15:54.734159 19064 leader_election.cc:288] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:54.734205 19067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:54.734370 19067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [term 1 LEADER]: Becoming Leader. State: Replica: 4c9772300fce40469e9bb333b4fc0fdc, State: Running, Role: LEADER
I20260928 23:15:54.734373 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:54.734474 19067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } }
I20260928 23:15:54.734817 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c9772300fce40469e9bb333b4fc0fdc. Latest consensus state: current_term: 1 leader_uuid: "4c9772300fce40469e9bb333b4fc0fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260928 23:15:54.734891 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c9772300fce40469e9bb333b4fc0fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c9772300fce40469e9bb333b4fc0fdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39185 } } }
I20260928 23:15:54.734959 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:54.734899 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:54.735131 19076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:54.735637 19076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:54.736580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 18997
I20260928 23:15:54.736688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:54.736877 392 external_mini_cluster.cc:1428] Setting key 40115f7560cd37b06945fcc2d0b92896
I20260928 23:15:54.737440 19076 catalog_manager.cc:1347] Generated new cluster ID: 77806eb506e44fbb902c19b3a235c625
I20260928 23:15:54.737491 19076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:54.742875 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.737321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.0.98.62:39185)
0928 23:15:54.737777 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.737786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.737819 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:15:54.737936 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.737943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.738007 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.738140 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.738375 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.739799 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.740799 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.741441 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.741568 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.742202 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.742206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.742207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:54.742411 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.742454 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:54.742468 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.742535 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.742598 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.742674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":60,"threads_started":1}
I20260928 23:15:54.743698 19076 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:54.744132 19076 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:54.750031 19076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: Generated new TSK 0
I20260928 23:15:54.750176 19076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:15:54.757413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:54.838245 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.838442 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.838470 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.839601 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.839637 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.839670 19088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:54.840955 19088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.841162 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.841447 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.846887 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.846902 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.847062 19088 server_base.cc:1034] running on GCE node
W20260928 23:15:54.846941 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.847345 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.847579 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.848728 19088 hybrid_clock.cc:648] HybridClock initialized: now 1790637354848716 us; error 27 us; skew 500 ppm
I20260928 23:15:54.849872 19088 webserver.cc:460] Webserver started at http://127.0.98.1:45205/ using document root <none> and password file <none>
I20260928 23:15:54.850086 19088 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.850127 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.850255 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.851258 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "73f3d5af60a94d5e88b28362caa1cdaa"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "b2f91bda4bb1e23cc28941d9694b2a7f"
server_key_iv: "06b2ae734ad106ba70eb53ca51a5be98"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.851572 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "73f3d5af60a94d5e88b28362caa1cdaa"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "b2f91bda4bb1e23cc28941d9694b2a7f"
server_key_iv: "06b2ae734ad106ba70eb53ca51a5be98"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.852768 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:54.853490 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.853626 19088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:54.853706 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "73f3d5af60a94d5e88b28362caa1cdaa"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "b2f91bda4bb1e23cc28941d9694b2a7f"
server_key_iv: "06b2ae734ad106ba70eb53ca51a5be98"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.853770 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:54.889330 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:54.889640 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:54.889770 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:54.889997 19088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:54.890336 19088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:54.890372 19088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.890400 19088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:54.890419 19088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.895717 19088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45487
I20260928 23:15:54.895761 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45487 every 8 connection(s)
I20260928 23:15:54.896066 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:54.900674 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.896752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37439 (local address 127.0.98.62:39185)
0928 23:15:54.896830 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:54.896832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:54.897021 (+ 189us) server_negotiation.cc:407] Connection header received
0928 23:15:54.897517 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:54.897518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:54.897530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:54.897579 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:54.898206 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.898698 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.899590 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.899763 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.899806 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.900481 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:54.900482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:54.900482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:54.900486 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:54.900499 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:54.900503 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:54.900538 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:54.900591 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:15:54.900617 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 23:15:54.900856 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:54.896735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.1:37439)
0928 23:15:54.896906 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:54.896941 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:54.897439 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:54.897641 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:54.897644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:54.897653 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:54.898134 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.898139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.898763 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.898763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.899530 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:54.899532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:54.899817 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:54.899817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:54.899925 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:54.900364 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:54.900364 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:54.900392 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:54.900415 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:54.900540 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:54.900543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:54.900545 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:54.900593 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:15:54.900667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 23:15:54.901337 19216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260928 23:15:54.901422 19216 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:54.901563 19216 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260928 23:15:54.902060 19028 ts_manager.cc:194] Registered new tserver with Master: 73f3d5af60a94d5e88b28362caa1cdaa (127.0.98.1:45487)
I20260928 23:15:54.902637 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37439
I20260928 23:15:54.903921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19088
I20260928 23:15:54.904012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:54.904173 392 external_mini_cluster.cc:1428] Setting key 98d331f0619bc816e8a36bf343610055
I20260928 23:15:54.904939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:54.983702 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:54.983898 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:54.983935 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:54.985072 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:54.985110 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:54.985150 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:54.986454 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:54.986688 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:54.986949 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:54.991596 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.991679 19219 server_base.cc:1034] running on GCE node
W20260928 23:15:54.991787 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:54.991606 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:54.992053 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:54.992277 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:54.993407 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790637354993394 us; error 27 us; skew 500 ppm
I20260928 23:15:54.994520 19219 webserver.cc:460] Webserver started at http://127.0.98.2:33275/ using document root <none> and password file <none>
I20260928 23:15:54.994721 19219 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:54.994766 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:54.994886 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:54.995898 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d2bf478c0254f9cac6a6c8c20077564"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6c95be3557de08f485eea57b20c57fbb"
server_key_iv: "d9b06004b79829dccbe4fee6b72c148b"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.996228 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d2bf478c0254f9cac6a6c8c20077564"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6c95be3557de08f485eea57b20c57fbb"
server_key_iv: "d9b06004b79829dccbe4fee6b72c148b"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.997459 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:15:54.998076 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:54.998219 19219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:54.998296 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "7d2bf478c0254f9cac6a6c8c20077564"
format_stamp: "Formatted at 2026-09-28 23:15:54 on dist-test-slave-6vgd"
server_key: "6c95be3557de08f485eea57b20c57fbb"
server_key_iv: "d9b06004b79829dccbe4fee6b72c148b"
server_key_version: "encryptionkey@0"
I20260928 23:15:54.998354 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:55.013502 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:55.013787 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:55.013907 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:55.014107 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:55.014405 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:55.014438 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.014467 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:55.014489 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.019876 19219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34099
I20260928 23:15:55.019945 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34099 every 8 connection(s)
I20260928 23:15:55.020236 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:55.024982 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.020875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41821 (local address 127.0.98.62:39185)
0928 23:15:55.020938 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.020940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.021161 (+ 221us) server_negotiation.cc:407] Connection header received
0928 23:15:55.021668 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.021668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.021677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.021732 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:55.022339 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.022818 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.023755 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.023995 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.024043 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.024766 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:55.024767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:55.024767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:55.024771 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:55.024785 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:55.024789 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:55.024830 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.024890 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.024913 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:15:55.025156 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.020873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.2:41821)
0928 23:15:55.021044 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.021083 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.021584 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.021785 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.021788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.021796 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:55.022280 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.022284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.022880 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.022880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.023643 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.023645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.024058 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.024059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.024169 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.024617 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:55.024618 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:55.024646 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:55.024669 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:55.024836 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:55.024839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:55.024842 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.024886 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.024967 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 23:15:55.025563 19347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260928 23:15:55.025669 19347 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:55.025837 19347 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260928 23:15:55.026132 19028 ts_manager.cc:194] Registered new tserver with Master: 7d2bf478c0254f9cac6a6c8c20077564 (127.0.98.2:34099)
I20260928 23:15:55.026474 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41821
I20260928 23:15:55.029083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19219
I20260928 23:15:55.029157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:55.029354 392 external_mini_cluster.cc:1428] Setting key 46bf941f7df422deafc48f510aef5591
I20260928 23:15:55.030143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--builtin_ntp_servers=127.0.98.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:55.107254 19350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:55.107438 19350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:55.107457 19350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:55.108502 19350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:55.108533 19350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:55.108556 19350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:55.109846 19350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:55.110020 19350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:55.110255 19350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:55.114738 19355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:55.114745 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:55.114883 19350 server_base.cc:1034] running on GCE node
W20260928 23:15:55.114791 19356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:55.115181 19350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:55.115407 19350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:55.116539 19350 hybrid_clock.cc:648] HybridClock initialized: now 1790637355116523 us; error 33 us; skew 500 ppm
I20260928 23:15:55.117750 19350 webserver.cc:460] Webserver started at http://127.0.98.3:34635/ using document root <none> and password file <none>
I20260928 23:15:55.117955 19350 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:55.118001 19350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:55.118120 19350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:55.119119 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "2e103d4830a146b196be463be6980247"
format_stamp: "Formatted at 2026-09-28 23:15:55 on dist-test-slave-6vgd"
server_key: "5ff4be1862990d193a65d61d42f7306c"
server_key_iv: "918593e4195c5e5f9d28e1aef8c0d448"
server_key_version: "encryptionkey@0"
I20260928 23:15:55.119437 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2e103d4830a146b196be463be6980247"
format_stamp: "Formatted at 2026-09-28 23:15:55 on dist-test-slave-6vgd"
server_key: "5ff4be1862990d193a65d61d42f7306c"
server_key_iv: "918593e4195c5e5f9d28e1aef8c0d448"
server_key_version: "encryptionkey@0"
I20260928 23:15:55.120590 19350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:15:55.121186 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.121331 19350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.121405 19350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "2e103d4830a146b196be463be6980247"
format_stamp: "Formatted at 2026-09-28 23:15:55 on dist-test-slave-6vgd"
server_key: "5ff4be1862990d193a65d61d42f7306c"
server_key_iv: "918593e4195c5e5f9d28e1aef8c0d448"
server_key_version: "encryptionkey@0"
I20260928 23:15:55.121464 19350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:55.131026 19350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:55.131278 19350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:55.131397 19350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:55.131587 19350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:55.131867 19350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:55.131898 19350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.131925 19350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:55.131948 19350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:55.137202 19350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39993
I20260928 23:15:55.137290 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39993 every 8 connection(s)
I20260928 23:15:55.137580 19350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:55.142102 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.138213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43947 (local address 127.0.98.62:39185)
0928 23:15:55.138257 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.138258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.138467 (+ 209us) server_negotiation.cc:407] Connection header received
0928 23:15:55.138908 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.138909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.138918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.138967 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:55.139559 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.140051 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.140936 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.141127 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.141177 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.141907 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:55.141908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:55.141909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:55.141912 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:55.141924 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:55.141928 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:55.141966 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.142021 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.142043 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:15:55.142272 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.138195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39185 (local address 127.0.98.3:43947)
0928 23:15:55.138344 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.138374 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.138835 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.139017 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.139021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.139028 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:55.139486 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.139490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.140114 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.140115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.140868 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.140870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.141185 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.141185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.141329 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.141790 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:55.141790 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:55.141817 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:55.141838 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:55.141968 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:55.141970 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:55.141972 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.142014 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.142088 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260928 23:15:55.142692 19478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39185
I20260928 23:15:55.142769 19478 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:55.142928 19478 heartbeater.cc:511] Master 127.0.98.62:39185 requested a full tablet report, sending...
I20260928 23:15:55.143244 19028 ts_manager.cc:194] Registered new tserver with Master: 2e103d4830a146b196be463be6980247 (127.0.98.3:39993)
I20260928 23:15:55.143599 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43947
I20260928 23:15:55.144253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19350
I20260928 23:15:55.144344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:55.144485 392 external_mini_cluster.cc:1428] Setting key 75de943248b32733104ffc3768dd1a46
I20260928 23:15:55.145174 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:55.148974 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.146383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.0.98.62:39185)
0928 23:15:55.146417 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.146419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.146525 (+ 106us) server_negotiation.cc:407] Connection header received
0928 23:15:55.146591 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.146592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.146601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.146643 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:55.146792 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.147241 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.147940 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.148111 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.148162 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.148728 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:55.148729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:55.148730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:55.148734 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:55.148746 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:55.148750 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:55.148821 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.148889 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.148898 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:15:55.150293 19028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:55.151757 19028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:55.157500 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.0.98.2:34099)
0928 23:15:55.154248 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.154252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.154262 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:55.154296 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.154298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.154312 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.154381 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:55.154511 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155569 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.156604 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157133 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157194 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157303 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.157385 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.157419 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":69,"threads_started":1}
I20260928 23:15:55.157516 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34099 (local address 127.0.0.1:51520)
0928 23:15:55.154014 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.154023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.154069 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.154390 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.154391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.154393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:55.154436 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.154438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155631 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.155632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.156538 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.156541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157194 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.157289 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157302 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.157372 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.157454 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260928 23:15:55.157768 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45487 (local address 127.0.0.1:40036)
0928 23:15:55.153739 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.153775 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.153835 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.154140 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.154143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.154145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:55.154205 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.154207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155503 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.155504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.156575 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.156577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157561 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.157619 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157622 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.157666 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.157699 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":30,"thread_start_us":54,"threads_started":1}
I20260928 23:15:55.157953 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39993 (local address 127.0.0.1:53502)
0928 23:15:55.154121 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.154127 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.154164 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.154306 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.154307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.154308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:55.154337 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.154338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155318 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.155319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.156561 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.156563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157559 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.157802 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157812 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.157860 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.157896 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260928 23:15:55.158035 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.0.98.1:45487)
0928 23:15:55.153986 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.153991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.154000 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:55.154035 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.154037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.154051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.154119 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:55.154273 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155428 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.156656 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157385 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157448 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157524 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.157923 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.157961 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260928 23:15:55.158213 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.153845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53502 (local address 127.0.98.3:39993)
0928 23:15:55.153986 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.153991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.154177 (+ 186us) server_negotiation.cc:407] Connection header received
0928 23:15:55.154220 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.154222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.154235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.154302 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:55.154529 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.155245 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.156657 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.157385 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.157448 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.157524 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.158101 (+ 577us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.158138 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260928 23:15:55.158246 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 (DEFAULT_TABLE table=test-table [id=6c72ce79d3e145db9eebfaff7a38c28d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:55.158613 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:55.158771 19412 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 (DEFAULT_TABLE table=test-table [id=6c72ce79d3e145db9eebfaff7a38c28d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:55.159075 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:55.161077 19492 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Bootstrap starting.
I20260928 23:15:55.161607 19493 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Bootstrap starting.
I20260928 23:15:55.161841 19492 tablet_bootstrap.cc:654] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:55.161845 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 (DEFAULT_TABLE table=test-table [id=6c72ce79d3e145db9eebfaff7a38c28d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:55.162153 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:55.162155 19492 log.cc:824] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:55.162180 19493 tablet_bootstrap.cc:654] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:55.162452 19493 log.cc:824] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:55.162906 19492 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: No bootstrap required, opened a new log
I20260928 23:15:55.162976 19492 ts_tablet_manager.cc:1397] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:55.163043 19493 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: No bootstrap required, opened a new log
I20260928 23:15:55.163102 19493 ts_tablet_manager.cc:1397] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:55.164563 19496 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Bootstrap starting.
I20260928 23:15:55.165266 19496 tablet_bootstrap.cc:654] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:55.165518 19496 log.cc:824] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:55.165432 19492 raft_consensus.cc:348] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.165432 19493 raft_consensus.cc:348] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.165583 19492 raft_consensus.cc:374] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:55.165587 19493 raft_consensus.cc:374] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:55.165627 19492 raft_consensus.cc:729] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d2bf478c0254f9cac6a6c8c20077564, State: Initialized, Role: FOLLOWER
I20260928 23:15:55.165627 19493 raft_consensus.cc:729] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e103d4830a146b196be463be6980247, State: Initialized, Role: FOLLOWER
I20260928 23:15:55.165736 19492 consensus_queue.cc:260] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.165736 19493 consensus_queue.cc:260] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.165984 19493 ts_tablet_manager.cc:1428] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:15:55.165984 19492 ts_tablet_manager.cc:1428] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:15:55.166013 19478 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
I20260928 23:15:55.166088 19347 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
I20260928 23:15:55.166608 19496 tablet_bootstrap.cc:492] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: No bootstrap required, opened a new log
I20260928 23:15:55.166668 19496 ts_tablet_manager.cc:1397] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:55.168454 19496 raft_consensus.cc:348] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.168579 19496 raft_consensus.cc:374] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:55.168612 19496 raft_consensus.cc:729] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f3d5af60a94d5e88b28362caa1cdaa, State: Initialized, Role: FOLLOWER
I20260928 23:15:55.168712 19496 consensus_queue.cc:260] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.168913 19496 ts_tablet_manager.cc:1428] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:55.168960 19216 heartbeater.cc:503] Master 127.0.98.62:39185 was elected leader, sending a full tablet report...
W20260928 23:15:55.270740 19348 tablet.cc:2369] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:15:55.388124 19479 tablet.cc:2369] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 23:15:55.396706 19217 tablet.cc:2369] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:55.457597 19499 raft_consensus.cc:482] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:55.457731 19499 raft_consensus.cc:504] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.458083 19499 leader_election.cc:288] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73f3d5af60a94d5e88b28362caa1cdaa (127.0.98.1:45487), 7d2bf478c0254f9cac6a6c8c20077564 (127.0.98.2:34099)
I20260928 23:15:55.461799 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.458344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46323 (local address 127.0.98.2:34099)
0928 23:15:55.458393 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.458395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.458528 (+ 133us) server_negotiation.cc:407] Connection header received
0928 23:15:55.458580 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.458581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.458590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.458636 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:55.458771 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.459473 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.460883 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.461447 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.461513 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.461520 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.461694 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.461731 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:15:55.461799 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.458344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33331 (local address 127.0.98.1:45487)
0928 23:15:55.458435 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.458437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.458446 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:55.458471 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.458471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.458480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.458551 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:55.458705 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.459450 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.460882 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.461450 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.461510 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.461520 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.461694 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.461730 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 23:15:55.461815 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.458264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45487 (local address 127.0.98.3:33331)
0928 23:15:55.458334 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.458347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.458392 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.458563 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.458564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.458566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:55.458635 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.458637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.459526 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.459527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.460797 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.460800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.461524 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.461525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.461629 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.461633 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.461683 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.461727 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":47}
I20260928 23:15:55.461824 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.458264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34099 (local address 127.0.98.3:46323)
0928 23:15:55.458468 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.458475 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:55.458528 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:55.458652 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:55.458653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:55.458655 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:55.458714 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.458716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.459545 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.459545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.460797 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:55.460800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.461543 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.461544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:55.461629 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.461633 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:55.461683 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:15:55.461727 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":86,"thread_start_us":110,"threads_started":1}
I20260928 23:15:55.462208 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1d3b80d1404d2e8dd3d5cd138cb6f9" candidate_uuid: "2e103d4830a146b196be463be6980247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" is_pre_election: true
I20260928 23:15:55.462234 19301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1d3b80d1404d2e8dd3d5cd138cb6f9" candidate_uuid: "2e103d4830a146b196be463be6980247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d2bf478c0254f9cac6a6c8c20077564" is_pre_election: true
I20260928 23:15:55.462374 19170 raft_consensus.cc:2454] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e103d4830a146b196be463be6980247 in term 0.
I20260928 23:15:55.462374 19301 raft_consensus.cc:2454] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e103d4830a146b196be463be6980247 in term 0.
I20260928 23:15:55.462591 19365 leader_election.cc:302] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e103d4830a146b196be463be6980247, 73f3d5af60a94d5e88b28362caa1cdaa; no voters:
I20260928 23:15:55.462754 19499 raft_consensus.cc:2781] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:55.462803 19499 raft_consensus.cc:482] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:55.462818 19499 raft_consensus.cc:3037] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:55.463443 19499 raft_consensus.cc:504] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.463564 19499 leader_election.cc:288] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [CANDIDATE]: Term 1 election: Requested vote from peers 73f3d5af60a94d5e88b28362caa1cdaa (127.0.98.1:45487), 7d2bf478c0254f9cac6a6c8c20077564 (127.0.98.2:34099)
I20260928 23:15:55.463716 19301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1d3b80d1404d2e8dd3d5cd138cb6f9" candidate_uuid: "2e103d4830a146b196be463be6980247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d2bf478c0254f9cac6a6c8c20077564"
I20260928 23:15:55.463721 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1d3b80d1404d2e8dd3d5cd138cb6f9" candidate_uuid: "2e103d4830a146b196be463be6980247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73f3d5af60a94d5e88b28362caa1cdaa"
I20260928 23:15:55.463783 19170 raft_consensus.cc:3037] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:55.463779 19301 raft_consensus.cc:3037] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:55.464443 19301 raft_consensus.cc:2454] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e103d4830a146b196be463be6980247 in term 1.
I20260928 23:15:55.464529 19170 raft_consensus.cc:2454] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e103d4830a146b196be463be6980247 in term 1.
I20260928 23:15:55.464604 19368 leader_election.cc:302] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e103d4830a146b196be463be6980247, 7d2bf478c0254f9cac6a6c8c20077564; no voters:
I20260928 23:15:55.464692 19499 raft_consensus.cc:2781] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:55.464821 19499 raft_consensus.cc:686] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 1 LEADER]: Becoming Leader. State: Replica: 2e103d4830a146b196be463be6980247, State: Running, Role: LEADER
I20260928 23:15:55.464892 19499 consensus_queue.cc:237] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } }
I20260928 23:15:55.465682 19027 catalog_manager.cc:5495] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e103d4830a146b196be463be6980247 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2e103d4830a146b196be463be6980247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e103d4830a146b196be463be6980247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39993 } health_report { overall_health: HEALTHY } } }
I20260928 23:15:55.509164 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.506936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.0.98.3:39993)
0928 23:15:55.506989 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:55.506991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:55.507000 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:55.507045 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:55.507046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:55.507069 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:55.507130 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:55.507254 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.507729 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.508457 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:55.508640 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:55.508691 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:55.508827 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:55.508923 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:55.509008 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:55.509054 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 23:15:55.509063 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 23:15:55.510862 19170 raft_consensus.cc:1261] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 73f3d5af60a94d5e88b28362caa1cdaa [term 1 FOLLOWER]: Refusing update from remote peer 2e103d4830a146b196be463be6980247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:55.510866 19301 raft_consensus.cc:1261] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 1 FOLLOWER]: Refusing update from remote peer 2e103d4830a146b196be463be6980247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:55.511188 19499 consensus_queue.cc:1035] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d2bf478c0254f9cac6a6c8c20077564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:55.511319 19502 consensus_queue.cc:1035] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73f3d5af60a94d5e88b28362caa1cdaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:55.513479 19505 mvcc.cc:205] Tried to move back new op lower bound from 7334450608170782720 to 7334450607985389568. Current Snapshot: MvccSnapshot[applied={T|T < 7334450608170782720}]
I20260928 23:15:55.514169 19506 mvcc.cc:205] Tried to move back new op lower bound from 7334450608170782720 to 7334450607985389568. Current Snapshot: MvccSnapshot[applied={T|T < 7334450608170782720}]
I20260928 23:15:55.515880 19028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:55.515966 19028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:55.516651 19028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c9772300fce40469e9bb333b4fc0fdc: Sending DeleteTablet for 3 replicas of tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9
I20260928 23:15:55.516933 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:55 UTC) from {username='slave'} at 127.0.0.1:53502
I20260928 23:15:55.516940 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:55 UTC) from {username='slave'} at 127.0.0.1:51520
I20260928 23:15:55.516993 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:55 UTC) from {username='slave'} at 127.0.0.1:40036
I20260928 23:15:55.517199 19514 tablet_replica.cc:330] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: stopping tablet replica
I20260928 23:15:55.517305 19514 raft_consensus.cc:2229] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:55.517390 19514 raft_consensus.cc:2258] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:55.517499 19515 tablet_replica.cc:330] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: stopping tablet replica
I20260928 23:15:55.517516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19088
I20260928 23:15:55.517621 19515 raft_consensus.cc:2229] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:55.517733 19515 raft_consensus.cc:2258] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:55.518679 19514 ts_tablet_manager.cc:1905] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:55.520349 19515 ts_tablet_manager.cc:1905] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:55.520413 19514 ts_tablet_manager.cc:1918] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:55.520503 19514 log.cc:1196] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/1e1d3b80d1404d2e8dd3d5cd138cb6f9
I20260928 23:15:55.520879 19514 ts_tablet_manager.cc:1939] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 2e103d4830a146b196be463be6980247: Deleting consensus metadata
I20260928 23:15:55.521355 19013 catalog_manager.cc:4841] TS 2e103d4830a146b196be463be6980247 (127.0.98.3:39993): tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 (table test-table [id=6c72ce79d3e145db9eebfaff7a38c28d]) successfully deleted
I20260928 23:15:55.521883 19515 ts_tablet_manager.cc:1918] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:55.521942 19515 log.cc:1196] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/1e1d3b80d1404d2e8dd3d5cd138cb6f9
I20260928 23:15:55.522131 19515 ts_tablet_manager.cc:1939] T 1e1d3b80d1404d2e8dd3d5cd138cb6f9 P 7d2bf478c0254f9cac6a6c8c20077564: Deleting consensus metadata
I20260928 23:15:55.522557 19015 catalog_manager.cc:4841] TS 7d2bf478c0254f9cac6a6c8c20077564 (127.0.98.2:34099): tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9 (table test-table [id=6c72ce79d3e145db9eebfaff7a38c28d]) successfully deleted
W20260928 23:15:55.524096 19012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45487 (error 108)
I20260928 23:15:55.524343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19219
I20260928 23:15:55.524492 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:55.524370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45487 (local address 127.0.0.1:40044)
0928 23:15:55.524416 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:55.524463 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:15:55.525007 19012 catalog_manager.cc:4584] TS 73f3d5af60a94d5e88b28362caa1cdaa (127.0.98.1:45487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e1d3b80d1404d2e8dd3d5cd138cb6f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45487: connect: Connection refused (error 111)
I20260928 23:15:55.530767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19350
I20260928 23:15:55.535007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 18997
2026-09-28T23:15:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:15:55 dist-test-slave-6vgd krb5kdc[19520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:15:55 dist-test-slave-6vgd krb5kdc[19520](info): set up 2 sockets
Sep 28 23:15:55 dist-test-slave-6vgd krb5kdc[19520](info): commencing operation
krb5kdc: starting...
W20260928 23:15:56.472815 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:15:56 dist-test-slave-6vgd krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:15:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:15:56.582342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:56.660359 19533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:56.660559 19533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:56.660579 19533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:56.661728 19533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:56.661772 19533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:56.661784 19533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:56.661808 19533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:56.661830 19533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:56.663278 19533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:56.663511 19533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:56.663725 19533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:56.668251 19539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:56.668292 19538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:56.668257 19541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:56.668455 19533 server_base.cc:1034] running on GCE node
I20260928 23:15:56.668666 19533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:56.668955 19533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:56.670123 19533 hybrid_clock.cc:648] HybridClock initialized: now 1790637356670106 us; error 31 us; skew 500 ppm
Sep 28 23:15:56 dist-test-slave-6vgd krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:56.672406 19533 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:15:56.672781 19533 webserver.cc:460] Webserver started at http://127.0.98.62:41871/ using document root <none> and password file <none>
I20260928 23:15:56.672973 19533 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:56.673013 19533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:56.673094 19533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:56.674124 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "977196f091af482c8dfe7aa1bbbb9712"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "2a5addea11739c55007f81d0889b07f9"
server_key_iv: "07d1e978bd4405d116186f30b6d8681a"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.674398 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "977196f091af482c8dfe7aa1bbbb9712"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "2a5addea11739c55007f81d0889b07f9"
server_key_iv: "07d1e978bd4405d116186f30b6d8681a"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.675539 19533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:15:56.676139 19548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:56.676297 19533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:56.676385 19533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "977196f091af482c8dfe7aa1bbbb9712"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "2a5addea11739c55007f81d0889b07f9"
server_key_iv: "07d1e978bd4405d116186f30b6d8681a"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.676443 19533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:56.699033 19533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:56.700702 19533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:56.700809 19533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:56.704442 19533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37419
I20260928 23:15:56.704484 19600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37419 every 8 connection(s)
I20260928 23:15:56.704787 19533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:56.705591 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:56.706521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19533
I20260928 23:15:56.706617 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:56.706794 392 external_mini_cluster.cc:1428] Setting key 0070f7c03b59b67f2a55abfaa2b12dd3
I20260928 23:15:56.708020 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: Bootstrap starting.
I20260928 23:15:56.708695 19601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:56.708985 19601 log.cc:824] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:56.709635 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: No bootstrap required, opened a new log
I20260928 23:15:56.711781 19601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260928 23:15:56.711956 19601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:56.712000 19601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977196f091af482c8dfe7aa1bbbb9712, State: Initialized, Role: FOLLOWER
I20260928 23:15:56.712131 19601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260928 23:15:56.712208 19601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:56.712245 19601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:56.712292 19601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:56.712890 19601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260928 23:15:56.712984 19601 leader_election.cc:302] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977196f091af482c8dfe7aa1bbbb9712; no voters:
I20260928 23:15:56.713135 19601 leader_election.cc:288] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:56.713163 19606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 1 FOLLOWER]: Leader election won for term 1
Sep 28 23:15:56 dist-test-slave-6vgd krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:56.713294 19606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [term 1 LEADER]: Becoming Leader. State: Replica: 977196f091af482c8dfe7aa1bbbb9712, State: Running, Role: LEADER
I20260928 23:15:56.713364 19606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260928 23:15:56.713390 19601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:56.713686 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977196f091af482c8dfe7aa1bbbb9712. Latest consensus state: current_term: 1 leader_uuid: "977196f091af482c8dfe7aa1bbbb9712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260928 23:15:56.713763 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:56.713701 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977196f091af482c8dfe7aa1bbbb9712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977196f091af482c8dfe7aa1bbbb9712" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260928 23:15:56.713907 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:56.714195 19611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:56.714704 19611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:56.716228 19611 catalog_manager.cc:1347] Generated new cluster ID: 8b623045c0234147a68f3a6f16d2c229
I20260928 23:15:56.716276 19611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:56.717373 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:56.707251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.0.98.62:37419)
0928 23:15:56.707397 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:56.707404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:56.707428 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:15:56.707838 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:56.707844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:56.707893 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:56.708005 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:56.708732 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.709749 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.710737 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.711111 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.711183 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:56.713849 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:56.713854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:56.713854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:56.713866 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:56.715825 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:56.716192 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:56.716194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:56.716195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:56.716318 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:56.716499 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:56.716499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:56.716500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:56.716760 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:56.716895 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:56.717015 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 23:15:56.717142 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260928 23:15:56.735116 19611 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:56.735692 19611 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:56.743033 19611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: Generated new TSK 0
I20260928 23:15:56.743202 19611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 23:15:56.771636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:56.850590 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:56.850800 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:56.850821 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:56.851958 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:56.851997 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:56.852027 19627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:56.853518 19627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:56.853739 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:56.853945 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:56.858518 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:56.858525 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:56.858525 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:56.858767 19627 server_base.cc:1034] running on GCE node
I20260928 23:15:56.858960 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:56.859144 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:56.860272 19627 hybrid_clock.cc:648] HybridClock initialized: now 1790637356860260 us; error 31 us; skew 500 ppm
Sep 28 23:15:56 dist-test-slave-6vgd krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:56.862663 19627 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 23:15:56.863065 19627 webserver.cc:460] Webserver started at http://127.0.98.1:45993/ using document root <none> and password file <none>
I20260928 23:15:56.863250 19627 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:56.863296 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:56.863418 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:56.864415 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "4650fa26f6674073917bfd59fbcf8771"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "30bf212907e6c3f78d5624d1359ead1b"
server_key_iv: "0475f4ca886528348be864d17b6a26da"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.864729 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4650fa26f6674073917bfd59fbcf8771"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "30bf212907e6c3f78d5624d1359ead1b"
server_key_iv: "0475f4ca886528348be864d17b6a26da"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.866001 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:56.866607 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:56.866750 19627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:56.866827 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "4650fa26f6674073917bfd59fbcf8771"
format_stamp: "Formatted at 2026-09-28 23:15:56 on dist-test-slave-6vgd"
server_key: "30bf212907e6c3f78d5624d1359ead1b"
server_key_iv: "0475f4ca886528348be864d17b6a26da"
server_key_version: "encryptionkey@0"
I20260928 23:15:56.866891 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:56.881283 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:56.883405 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:56.883548 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:56.883762 19627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:56.884055 19627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:56.884087 19627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:56.884117 19627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:56.884137 19627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:56.889530 19627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37381
I20260928 23:15:56.889572 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37381 every 8 connection(s)
I20260928 23:15:56.889875 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 23:15:56 dist-test-slave-6vgd krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:56.895961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19627
I20260928 23:15:56.896061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:56.896229 392 external_mini_cluster.cc:1428] Setting key 1a950b032dcce9dda77c0efb1fb48731
I20260928 23:15:56.898689 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:56.890535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42489 (local address 127.0.98.62:37419)
0928 23:15:56.890577 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:56.890579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:56.890764 (+ 185us) server_negotiation.cc:407] Connection header received
0928 23:15:56.891714 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:56.891715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:56.891728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:56.891780 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:56.892391 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.892894 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.893915 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.894110 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.894164 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:56.896062 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:56.896064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:56.896064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:56.896072 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:56.897470 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:56.897833 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:56.897835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:56.897835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:56.897889 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:56.898094 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:56.898095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:56.898095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:56.898203 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:56.898264 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:56.898514 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 23:15:56.898600 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:15:56.898862 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:56.890514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:42489)
0928 23:15:56.890651 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:56.890686 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:56.891607 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:56.891855 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:56.891859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:56.892116 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:56.892306 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:56.892311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.892974 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.892975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:56.893839 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:56.893841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:56.894176 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:56.894176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:56.894291 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:56.894696 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:56.894704 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:56.895979 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:56.897539 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:56.897546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:56.897548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:56.897771 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:56.897939 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:56.897940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:56.897940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:56.898042 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:56.898253 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:56.898259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:56.898399 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 23:15:56.898489 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 23:15:56.898639 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260928 23:15:56.899323 19756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260928 23:15:56.899417 19756 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:56.899583 19756 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260928 23:15:56.900128 19565 ts_manager.cc:194] Registered new tserver with Master: 4650fa26f6674073917bfd59fbcf8771 (127.0.98.1:37381)
I20260928 23:15:56.901041 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42489
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 23:15:56.918006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:56.996044 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:56.996248 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:56.996270 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:56.997370 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:56.997406 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:56.997430 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:56.998839 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:56.999012 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:56.999183 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:57.003705 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.003865 19766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.003980 19761 server_base.cc:1034] running on GCE node
W20260928 23:15:57.003983 19769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.004215 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.004436 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.005564 19761 hybrid_clock.cc:648] HybridClock initialized: now 1790637357005551 us; error 26 us; skew 500 ppm
Sep 28 23:15:57 dist-test-slave-6vgd krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:57.007840 19761 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 23:15:57.008262 19761 webserver.cc:460] Webserver started at http://127.0.98.2:34179/ using document root <none> and password file <none>
I20260928 23:15:57.008446 19761 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.008492 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.008612 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.009645 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "4482cbfc57d34006a12da1aafffce7e3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "502d98c37b1048a11cf960724d9af776"
server_key_iv: "6bf96f84cc752a58509ab635bb4afcda"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.009960 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4482cbfc57d34006a12da1aafffce7e3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "502d98c37b1048a11cf960724d9af776"
server_key_iv: "6bf96f84cc752a58509ab635bb4afcda"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.011088 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:15:57.011690 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.011840 19761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:57.011917 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "4482cbfc57d34006a12da1aafffce7e3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "502d98c37b1048a11cf960724d9af776"
server_key_iv: "6bf96f84cc752a58509ab635bb4afcda"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.011976 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.027704 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.029805 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.029945 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.030165 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:57.030457 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:57.030491 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.030519 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:57.030542 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.036019 19761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39059
I20260928 23:15:57.036084 19889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39059 every 8 connection(s)
I20260928 23:15:57.036362 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:57.042276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19761
Sep 28 23:15:57 dist-test-slave-6vgd krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:57.042395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:57.042565 392 external_mini_cluster.cc:1428] Setting key 7a07b2e9513a628b36d34a5867b0dd5c
I20260928 23:15:57.045387 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.037028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38505 (local address 127.0.98.62:37419)
0928 23:15:57.037095 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.037097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.037289 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:15:57.038256 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.038258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.038267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.038323 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.038966 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.039470 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.040486 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.040667 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.040722 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.042875 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.042877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.042877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:57.042884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.044141 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.044575 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.044576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.044577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.044636 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.044831 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.044832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.044832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.044950 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.045008 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.045245 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.045321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 23:15:57.045588 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.036981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:38505)
0928 23:15:57.037160 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.037197 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.038144 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.038391 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.038394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.038669 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:57.038883 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.038889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.039557 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.039558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.040411 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.040414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.040764 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.040766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.040877 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.041351 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:57.041359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:57.042791 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:57.044210 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:57.044218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:57.044220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:57.044514 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:57.044693 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:57.044693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:57.044694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:57.044785 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:57.044999 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:57.045002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:57.045111 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.045185 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.045385 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260928 23:15:57.046027 19890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260928 23:15:57.046113 19890 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:57.046280 19890 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260928 23:15:57.046610 19565 ts_manager.cc:194] Registered new tserver with Master: 4482cbfc57d34006a12da1aafffce7e3 (127.0.98.2:39059)
I20260928 23:15:57.046964 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38505
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 23:15:57.065354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:57.142552 19895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:57.142751 19895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:57.142773 19895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:57.143884 19895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:57.143924 19895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:57.143952 19895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:57.145417 19895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:57.145622 19895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:57.145828 19895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:57.150375 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.150390 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.150385 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.150418 19895 server_base.cc:1034] running on GCE node
I20260928 23:15:57.150803 19895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.151017 19895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.152148 19895 hybrid_clock.cc:648] HybridClock initialized: now 1790637357152136 us; error 28 us; skew 500 ppm
Sep 28 23:15:57 dist-test-slave-6vgd krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:57.154484 19895 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 23:15:57.154860 19895 webserver.cc:460] Webserver started at http://127.0.98.3:43313/ using document root <none> and password file <none>
I20260928 23:15:57.155026 19895 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.155063 19895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.155149 19895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.156095 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "aad3880b4f2e48fa944cb8915894bf45"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "cf5cee59e1474b932f96a230c8f34abe"
server_key_iv: "f20c659159b4b11f6603d213b85ac2f8"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.156379 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aad3880b4f2e48fa944cb8915894bf45"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "cf5cee59e1474b932f96a230c8f34abe"
server_key_iv: "f20c659159b4b11f6603d213b85ac2f8"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.157580 19895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:15:57.158183 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.158319 19895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:15:57.158392 19895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "aad3880b4f2e48fa944cb8915894bf45"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "cf5cee59e1474b932f96a230c8f34abe"
server_key_iv: "f20c659159b4b11f6603d213b85ac2f8"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.158454 19895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.199996 19895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.202167 19895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.202286 19895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.202517 19895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:57.202800 19895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:57.202831 19895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.202854 19895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:57.202876 19895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.208025 19895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35841
I20260928 23:15:57.208091 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35841 every 8 connection(s)
I20260928 23:15:57.208351 19895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:57.209784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 19895
I20260928 23:15:57.209903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:57.210094 392 external_mini_cluster.cc:1428] Setting key e576c473cb6d61b905bc881ae2d96094
Sep 28 23:15:57 dist-test-slave-6vgd krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:15:57.216979 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.208989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40319 (local address 127.0.98.62:37419)
0928 23:15:57.209033 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.209035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.209252 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:15:57.210329 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.210331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.210343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.210400 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.211101 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.211583 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.212477 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.212653 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.212695 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.214818 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.214819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.214819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:57.214825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.215932 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.216253 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.216254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.216254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.216298 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.216461 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.216462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.216462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.216566 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.216611 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.216869 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.216918 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:15:57.217149 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.208968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:40319)
0928 23:15:57.209116 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.209155 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.210228 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.210469 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.210473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.210793 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:57.211022 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.211026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.211645 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.211646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.212419 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.212421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.212714 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.212715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.212817 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.213295 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:15:57.213305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:57.214745 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:57.216004 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:57.216009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:57.216011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:57.216201 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:57.216340 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:15:57.216340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:15:57.216341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:15:57.216422 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:15:57.216637 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:57.216642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:57.216766 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.216854 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.216964 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 23:15:57.217820 20024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260928 23:15:57.217900 20024 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:57.218055 20024 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260928 23:15:57.218366 19565 ts_manager.cc:194] Registered new tserver with Master: aad3880b4f2e48fa944cb8915894bf45 (127.0.98.3:35841)
I20260928 23:15:57.218719 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40319
I20260928 23:15:57.221493 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:57.227506 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.222623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.0.98.62:37419)
0928 23:15:57.222687 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.222688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.222739 (+ 51us) server_negotiation.cc:407] Connection header received
0928 23:15:57.222797 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.222798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.222806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.222849 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.223319 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.223774 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.224440 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.224631 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.224688 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.225638 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.225640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.225640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:57.225646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.226568 (+ 922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.226842 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.226843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.226843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.226893 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:57.227048 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:57.227048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:57.227049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:57.227178 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.227264 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.227359 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.227426 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 23:15:57.228647 19565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:57.230201 19565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:57.236254 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.232165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.0.98.2:39059)
0928 23:15:57.232300 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.232304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.232783 (+ 479us) server_negotiation.cc:407] Connection header received
0928 23:15:57.232828 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.232830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.232846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.232897 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.233039 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.233746 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235185 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.235898 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235968 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.236026 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.236145 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.236182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 23:15:57.236241 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.231917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37381 (local address 127.0.0.1:44362)
0928 23:15:57.232062 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.232095 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.232154 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.232653 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.232656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.232658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.232715 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.232716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.234007 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.234008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.235099 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.235102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.235984 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.236048 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.236066 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.236100 (+ 34us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.236150 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260928 23:15:57.236248 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.231918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39059 (local address 127.0.0.1:58924)
0928 23:15:57.232574 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.232580 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.232618 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.232937 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.232938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.232940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.232972 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.232973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.233827 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.233828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.235112 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.235114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.235968 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.236044 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.236058 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.236122 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.236166 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":31,"thread_start_us":50,"threads_started":1}
I20260928 23:15:57.236557 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.232354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.98.1:37381)
0928 23:15:57.232489 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.232491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.232499 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:15:57.232538 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.232540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.232555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.232635 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.232779 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.233910 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235183 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.235898 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.236357 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.236423 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.236448 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.236487 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 23:15:57.237018 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 292c34954346447e9a9d37c5d019cb6c (DEFAULT_TABLE table=test-table [id=5ed6bb8484c940b5bcca813539ab25f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.237406 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292c34954346447e9a9d37c5d019cb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.237385 19690 tablet_service.cc:1467] Processing CreateTablet for tablet 292c34954346447e9a9d37c5d019cb6c (DEFAULT_TABLE table=test-table [id=5ed6bb8484c940b5bcca813539ab25f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.237730 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292c34954346447e9a9d37c5d019cb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.237819 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.231918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.0.1:33690)
0928 23:15:57.232477 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.232484 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.232532 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.232748 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.232749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.232751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.232800 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.232802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.235189 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.235854 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.235856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.237449 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.237521 (+ 72us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.237616 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.237620 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.237684 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.237721 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260928 23:15:57.237833 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.232104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.3:35841)
0928 23:15:57.232236 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.232240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.232539 (+ 299us) server_negotiation.cc:407] Connection header received
0928 23:15:57.232591 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.232593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.232605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.232677 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.232870 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.233928 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.235934 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.237368 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.237446 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.237511 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.237699 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.237736 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260928 23:15:57.238441 19958 tablet_service.cc:1467] Processing CreateTablet for tablet 292c34954346447e9a9d37c5d019cb6c (DEFAULT_TABLE table=test-table [id=5ed6bb8484c940b5bcca813539ab25f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.238765 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292c34954346447e9a9d37c5d019cb6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.240314 20038 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: Bootstrap starting.
I20260928 23:15:57.240680 20039 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Bootstrap starting.
I20260928 23:15:57.240960 20038 tablet_bootstrap.cc:654] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.241329 20039 tablet_bootstrap.cc:654] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.241470 20038 log.cc:824] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.241830 20039 log.cc:824] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.241845 20041 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Bootstrap starting.
I20260928 23:15:57.242161 20038 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: No bootstrap required, opened a new log
I20260928 23:15:57.242216 20038 ts_tablet_manager.cc:1397] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:57.242419 20039 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: No bootstrap required, opened a new log
I20260928 23:15:57.242426 20041 tablet_bootstrap.cc:654] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.242460 20039 ts_tablet_manager.cc:1397] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:15:57.242924 20041 log.cc:824] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.243909 20038 raft_consensus.cc:348] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.243901 20039 raft_consensus.cc:348] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.244027 20038 raft_consensus.cc:374] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.244028 20039 raft_consensus.cc:374] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.244050 20038 raft_consensus.cc:729] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4650fa26f6674073917bfd59fbcf8771, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.244050 20039 raft_consensus.cc:729] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4482cbfc57d34006a12da1aafffce7e3, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.244140 20038 consensus_queue.cc:260] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.244145 20039 consensus_queue.cc:260] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.244366 20039 ts_tablet_manager.cc:1428] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 23:15:57.244393 20038 ts_tablet_manager.cc:1428] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:57.244436 19756 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
I20260928 23:15:57.244470 19890 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
I20260928 23:15:57.245268 20041 tablet_bootstrap.cc:492] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: No bootstrap required, opened a new log
I20260928 23:15:57.245361 20041 ts_tablet_manager.cc:1397] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 23:15:57.246721 20041 raft_consensus.cc:348] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.246847 20041 raft_consensus.cc:374] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.246869 20041 raft_consensus.cc:729] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad3880b4f2e48fa944cb8915894bf45, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.246960 20041 consensus_queue.cc:260] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.247198 20041 ts_tablet_manager.cc:1428] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 23:15:57.247210 20024 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
W20260928 23:15:57.286911 19891 tablet.cc:2369] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:57.323731 20045 raft_consensus.cc:482] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:57.323861 20045 raft_consensus.cc:504] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.324225 20045 leader_election.cc:288] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4482cbfc57d34006a12da1aafffce7e3 (127.0.98.2:39059), aad3880b4f2e48fa944cb8915894bf45 (127.0.98.3:35841)
I20260928 23:15:57.327801 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.324381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39059 (local address 127.0.98.1:54429)
0928 23:15:57.324430 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.324441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.324501 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.324632 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.324633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.324635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.324707 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.324709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.325600 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.325600 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.326798 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.326800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.327498 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.327499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.327600 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.327605 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.327661 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.327708 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":13}
I20260928 23:15:57.327809 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.324494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59227 (local address 127.0.98.3:35841)
0928 23:15:57.324532 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.324534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.324680 (+ 146us) server_negotiation.cc:407] Connection header received
0928 23:15:57.324709 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.324710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.324717 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.324764 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.324878 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.325575 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.326875 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.327427 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.327493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.327502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.327697 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.327740 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:15:57.327809 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.324469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54429 (local address 127.0.98.2:39059)
0928 23:15:57.324512 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.324514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.324523 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:57.324551 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.324552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.324563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.324626 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.324771 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.325532 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.326875 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.327423 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.327493 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.327502 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.327697 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.327740 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:15:57.327805 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.324381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35841 (local address 127.0.98.1:59227)
0928 23:15:57.324612 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.324621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.324670 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.324775 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.324775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.324777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.324827 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.324828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.325635 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.325636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.326798 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.326800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.327500 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.327501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.327609 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.327611 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.327661 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.327708 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":35,"thread_start_us":112,"threads_started":1}
I20260928 23:15:57.328171 19978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292c34954346447e9a9d37c5d019cb6c" candidate_uuid: "4650fa26f6674073917bfd59fbcf8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad3880b4f2e48fa944cb8915894bf45" is_pre_election: true
I20260928 23:15:57.328186 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292c34954346447e9a9d37c5d019cb6c" candidate_uuid: "4650fa26f6674073917bfd59fbcf8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482cbfc57d34006a12da1aafffce7e3" is_pre_election: true
I20260928 23:15:57.328308 19978 raft_consensus.cc:2454] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650fa26f6674073917bfd59fbcf8771 in term 0.
I20260928 23:15:57.328320 19844 raft_consensus.cc:2454] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4650fa26f6674073917bfd59fbcf8771 in term 0.
I20260928 23:15:57.328521 19646 leader_election.cc:302] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4482cbfc57d34006a12da1aafffce7e3, 4650fa26f6674073917bfd59fbcf8771; no voters:
I20260928 23:15:57.328691 20045 raft_consensus.cc:2781] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:57.328748 20045 raft_consensus.cc:482] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:57.328764 20045 raft_consensus.cc:3037] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:57.329487 20045 raft_consensus.cc:504] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.329599 20045 leader_election.cc:288] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [CANDIDATE]: Term 1 election: Requested vote from peers 4482cbfc57d34006a12da1aafffce7e3 (127.0.98.2:39059), aad3880b4f2e48fa944cb8915894bf45 (127.0.98.3:35841)
I20260928 23:15:57.329733 19978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292c34954346447e9a9d37c5d019cb6c" candidate_uuid: "4650fa26f6674073917bfd59fbcf8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad3880b4f2e48fa944cb8915894bf45"
I20260928 23:15:57.329746 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292c34954346447e9a9d37c5d019cb6c" candidate_uuid: "4650fa26f6674073917bfd59fbcf8771" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482cbfc57d34006a12da1aafffce7e3"
I20260928 23:15:57.329807 19978 raft_consensus.cc:3037] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:57.329809 19844 raft_consensus.cc:3037] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:57.330449 19844 raft_consensus.cc:2454] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4650fa26f6674073917bfd59fbcf8771 in term 1.
I20260928 23:15:57.330482 19978 raft_consensus.cc:2454] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4650fa26f6674073917bfd59fbcf8771 in term 1.
I20260928 23:15:57.330616 19643 leader_election.cc:302] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4650fa26f6674073917bfd59fbcf8771, aad3880b4f2e48fa944cb8915894bf45; no voters:
I20260928 23:15:57.330698 20045 raft_consensus.cc:2781] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:57.330821 20045 raft_consensus.cc:686] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 1 LEADER]: Becoming Leader. State: Replica: 4650fa26f6674073917bfd59fbcf8771, State: Running, Role: LEADER
I20260928 23:15:57.330893 20045 consensus_queue.cc:237] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } }
I20260928 23:15:57.331673 19564 catalog_manager.cc:5495] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4650fa26f6674073917bfd59fbcf8771 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4650fa26f6674073917bfd59fbcf8771" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4650fa26f6674073917bfd59fbcf8771" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37381 } health_report { overall_health: HEALTHY } } }
I20260928 23:15:57.349812 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.347788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.98.1:37381)
0928 23:15:57.347831 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.347832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.347850 (+ 18us) server_negotiation.cc:407] Connection header received
0928 23:15:57.347880 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.347881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.347900 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.347947 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:57.348045 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.348488 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.349179 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.349383 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.349424 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.349535 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:57.349642 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:57.349696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.349708 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.349723 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:15:57.351439 19978 raft_consensus.cc:1261] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 1 FOLLOWER]: Refusing update from remote peer 4650fa26f6674073917bfd59fbcf8771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:57.351439 19844 raft_consensus.cc:1261] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 1 FOLLOWER]: Refusing update from remote peer 4650fa26f6674073917bfd59fbcf8771: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:57.351725 20048 consensus_queue.cc:1035] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482cbfc57d34006a12da1aafffce7e3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:57.351845 20045 consensus_queue.cc:1035] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad3880b4f2e48fa944cb8915894bf45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:57.354405 20053 mvcc.cc:205] Tried to move back new op lower bound from 7334450615710138368 to 7334450615628808192. Current Snapshot: MvccSnapshot[applied={T|T < 7334450615710138368}]
I20260928 23:15:57.355051 20051 mvcc.cc:205] Tried to move back new op lower bound from 7334450615710138368 to 7334450615628808192. Current Snapshot: MvccSnapshot[applied={T|T < 7334450615710138368 or (T in {7334450615710138368})}]
I20260928 23:15:57.356227 19564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:57.356309 19564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:57.356949 19564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 977196f091af482c8dfe7aa1bbbb9712: Sending DeleteTablet for 3 replicas of tablet 292c34954346447e9a9d37c5d019cb6c
I20260928 23:15:57.357313 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet 292c34954346447e9a9d37c5d019cb6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44362
I20260928 23:15:57.357313 19958 tablet_service.cc:1514] Processing DeleteTablet for tablet 292c34954346447e9a9d37c5d019cb6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33690
I20260928 23:15:57.357317 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 292c34954346447e9a9d37c5d019cb6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58924
I20260928 23:15:57.357545 20062 tablet_replica.cc:330] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: stopping tablet replica
I20260928 23:15:57.357712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19627
I20260928 23:15:57.357740 20061 tablet_replica.cc:330] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771: stopping tablet replica
I20260928 23:15:57.357824 20061 raft_consensus.cc:2229] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:57.357903 20061 raft_consensus.cc:2258] T 292c34954346447e9a9d37c5d019cb6c P 4650fa26f6674073917bfd59fbcf8771 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:57.358031 20060 tablet_replica.cc:330] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: stopping tablet replica
I20260928 23:15:57.358194 20060 raft_consensus.cc:2229] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:57.358269 20060 raft_consensus.cc:2258] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:57.358306 20062 raft_consensus.cc:2229] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:57.358408 20062 raft_consensus.cc:2258] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:57.358642 20060 ts_tablet_manager.cc:1905] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:57.358767 20062 ts_tablet_manager.cc:1905] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:57.359999 20060 ts_tablet_manager.cc:1918] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:57.360052 20060 log.cc:1196] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/292c34954346447e9a9d37c5d019cb6c
I20260928 23:15:57.360157 20062 ts_tablet_manager.cc:1918] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:57.360213 20062 log.cc:1196] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/292c34954346447e9a9d37c5d019cb6c
I20260928 23:15:57.360249 20060 ts_tablet_manager.cc:1939] T 292c34954346447e9a9d37c5d019cb6c P aad3880b4f2e48fa944cb8915894bf45: Deleting consensus metadata
I20260928 23:15:57.360469 20062 ts_tablet_manager.cc:1939] T 292c34954346447e9a9d37c5d019cb6c P 4482cbfc57d34006a12da1aafffce7e3: Deleting consensus metadata
I20260928 23:15:57.360584 19549 catalog_manager.cc:4841] TS aad3880b4f2e48fa944cb8915894bf45 (127.0.98.3:35841): tablet 292c34954346447e9a9d37c5d019cb6c (table test-table [id=5ed6bb8484c940b5bcca813539ab25f0]) successfully deleted
I20260928 23:15:57.360788 19552 catalog_manager.cc:4841] TS 4482cbfc57d34006a12da1aafffce7e3 (127.0.98.2:39059): tablet 292c34954346447e9a9d37c5d019cb6c (table test-table [id=5ed6bb8484c940b5bcca813539ab25f0]) successfully deleted
W20260928 23:15:57.363867 19550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37381 (error 108)
I20260928 23:15:57.364049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19761
I20260928 23:15:57.364518 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.364132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37381 (local address 127.0.0.1:44376)
0928 23:15:57.364458 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.364488 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
W20260928 23:15:57.364742 19550 catalog_manager.cc:4584] TS 4650fa26f6674073917bfd59fbcf8771 (127.0.98.1:37381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 292c34954346447e9a9d37c5d019cb6c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37381: connect: Connection refused (error 111)
I20260928 23:15:57.367825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19895
I20260928 23:15:57.371610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 19533
2026-09-28T23:15:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T23:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:57Z Disabled control of system clock
I20260928 23:15:57.396270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40339
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:57.473707 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:57.473910 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:57.473928 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:57.474987 20067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:57.475018 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:57.475029 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:57.475040 20067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:57.475056 20067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:57.476301 20067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:57.476497 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:57.476724 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 23:15:57.481266 20067 server_base.cc:1034] running on GCE node
W20260928 23:15:57.481185 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.481189 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.481244 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.481655 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.481915 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.483043 20067 hybrid_clock.cc:648] HybridClock initialized: now 1790637357483027 us; error 26 us; skew 500 ppm
I20260928 23:15:57.484107 20067 webserver.cc:460] Webserver started at http://127.0.98.62:37459/ using document root <none> and password file <none>
I20260928 23:15:57.484329 20067 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.484373 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.484493 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.485525 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "1857e696e0c64efbb170dcb51788e9c4"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "48009676cb90a531832b6cf3cc496d43"
server_key_iv: "af5a68f9ee49757a2017ff3d46bb5426"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.485836 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "1857e696e0c64efbb170dcb51788e9c4"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "48009676cb90a531832b6cf3cc496d43"
server_key_iv: "af5a68f9ee49757a2017ff3d46bb5426"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.487090 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 23:15:57.487728 20081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.487875 20067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:57.487950 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "1857e696e0c64efbb170dcb51788e9c4"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "48009676cb90a531832b6cf3cc496d43"
server_key_iv: "af5a68f9ee49757a2017ff3d46bb5426"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.488014 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.500840 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.501106 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.501250 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.504776 20067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40339
I20260928 23:15:57.504809 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40339 every 8 connection(s)
I20260928 23:15:57.505105 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:57.506012 20134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.508077 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: Bootstrap starting.
I20260928 23:15:57.508554 20134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.508773 20134 log.cc:824] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.509346 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: No bootstrap required, opened a new log
I20260928 23:15:57.509891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20067
I20260928 23:15:57.509989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:57.510159 392 external_mini_cluster.cc:1428] Setting key 622abc5ce1ba8f1ba90146d9e6634769
I20260928 23:15:57.511193 20134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } }
I20260928 23:15:57.511350 20134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.511384 20134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1857e696e0c64efbb170dcb51788e9c4, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.511499 20134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } }
I20260928 23:15:57.511576 20134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:57.511605 20134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:57.511650 20134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:57.512253 20134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } }
I20260928 23:15:57.512356 20134 leader_election.cc:302] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1857e696e0c64efbb170dcb51788e9c4; no voters:
I20260928 23:15:57.512501 20134 leader_election.cc:288] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:57.512552 20139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:57.512691 20139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [term 1 LEADER]: Becoming Leader. State: Replica: 1857e696e0c64efbb170dcb51788e9c4, State: Running, Role: LEADER
I20260928 23:15:57.512722 20134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:57.512774 20139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } }
I20260928 23:15:57.513094 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1857e696e0c64efbb170dcb51788e9c4. Latest consensus state: current_term: 1 leader_uuid: "1857e696e0c64efbb170dcb51788e9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } } }
I20260928 23:15:57.513159 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:57.513146 20140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1857e696e0c64efbb170dcb51788e9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1857e696e0c64efbb170dcb51788e9c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40339 } } }
I20260928 23:15:57.513195 20140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:57.513449 20146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:57.513933 20146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:57.515468 20146 catalog_manager.cc:1347] Generated new cluster ID: 45ac1e40c102480a803ab5a0e2a4ed53
I20260928 23:15:57.515822 20146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:15:57.516536 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.510632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.0.98.62:40339)
0928 23:15:57.510850 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.510859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.510893 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:15:57.511170 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.511178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.511248 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.511391 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.511570 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.513240 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.514382 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.514888 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.515002 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.515899 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.515903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.515903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:57.516067 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.516103 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:57.516113 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.516168 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.516249 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.516326 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260928 23:15:57.560508 20146 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:57.561049 20146 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:57.564961 20146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: Generated new TSK 0
I20260928 23:15:57.565148 20146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:15:57.576944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--builtin_ntp_servers=127.0.98.20:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:57.659102 20158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:57.659304 20158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:57.659323 20158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:57.660475 20158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:57.660516 20158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:57.660542 20158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 23:15:57.661906 20158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:57.662132 20158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:57.662398 20158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:57.667178 20164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.667156 20166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.667135 20163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.667382 20158 server_base.cc:1034] running on GCE node
I20260928 23:15:57.667649 20158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.667879 20158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.669034 20158 hybrid_clock.cc:648] HybridClock initialized: now 1790637357669023 us; error 29 us; skew 500 ppm
I20260928 23:15:57.670325 20158 webserver.cc:460] Webserver started at http://127.0.98.1:38005/ using document root <none> and password file <none>
I20260928 23:15:57.670539 20158 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.670586 20158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.670709 20158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.671773 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data/instance:
uuid: "3aacc6dee2c144f98a45533e25cf93d3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "548d3a309bf8e2f77b4b883632c661ea"
server_key_iv: "a85c51a51e8e8c29265b70678a084c02"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.672101 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3aacc6dee2c144f98a45533e25cf93d3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "548d3a309bf8e2f77b4b883632c661ea"
server_key_iv: "a85c51a51e8e8c29265b70678a084c02"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.673318 20158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:15:57.673972 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.674117 20158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:57.674192 20158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal
uuid: "3aacc6dee2c144f98a45533e25cf93d3"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "548d3a309bf8e2f77b4b883632c661ea"
server_key_iv: "a85c51a51e8e8c29265b70678a084c02"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.674257 20158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.695735 20158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.696023 20158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.696151 20158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.696376 20158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:57.696691 20158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:57.696723 20158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.696753 20158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:57.696774 20158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.702257 20158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38605
I20260928 23:15:57.702314 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38605 every 8 connection(s)
I20260928 23:15:57.702633 20158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/data/info.pb
I20260928 23:15:57.707293 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.703340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50631 (local address 127.0.98.62:40339)
0928 23:15:57.703396 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.703398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.703601 (+ 203us) server_negotiation.cc:407] Connection header received
0928 23:15:57.704075 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.704076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.704085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.704145 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.704765 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.705299 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.706201 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.706378 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.706420 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.707098 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.707099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.707100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:57.707104 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.707117 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:57.707120 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.707158 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.707212 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.707233 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:15:57.707480 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.703291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40339 (local address 127.0.98.1:50631)
0928 23:15:57.703479 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.703521 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.703999 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.704199 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.704202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.704211 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:57.704704 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.704709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.705365 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.705366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.706140 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.706142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.706428 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.706429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.706528 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.706979 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:57.706979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:57.707008 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:57.707029 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:57.707158 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:57.707161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:57.707163 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.707206 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.707279 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 23:15:57.707954 20286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40339
I20260928 23:15:57.708036 20286 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:57.708205 20286 heartbeater.cc:511] Master 127.0.98.62:40339 requested a full tablet report, sending...
I20260928 23:15:57.708670 20098 ts_manager.cc:194] Registered new tserver with Master: 3aacc6dee2c144f98a45533e25cf93d3 (127.0.98.1:38605)
I20260928 23:15:57.709267 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50631
I20260928 23:15:57.711766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20158
I20260928 23:15:57.711853 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-0/wal/instance
I20260928 23:15:57.712009 392 external_mini_cluster.cc:1428] Setting key 7ea7101ab1d2c8dd5161a21c18ec4bc0
I20260928 23:15:57.712795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--builtin_ntp_servers=127.0.98.20:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 23:15:57.790666 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:57.790874 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:57.790894 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:57.791997 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:57.792032 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:57.792055 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 23:15:57.793342 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:57.793525 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:57.793771 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:57.798436 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.798457 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.798699 20289 server_base.cc:1034] running on GCE node
W20260928 23:15:57.798733 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.798945 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.799163 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.800323 20289 hybrid_clock.cc:648] HybridClock initialized: now 1790637357800310 us; error 29 us; skew 500 ppm
I20260928 23:15:57.801604 20289 webserver.cc:460] Webserver started at http://127.0.98.2:44321/ using document root <none> and password file <none>
I20260928 23:15:57.801837 20289 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.801887 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.802004 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.803035 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca0e0c169040479ba95437ca504e819e"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "85dbd7e42e09c31b656c01d53a7c115a"
server_key_iv: "4eab5dedf48ddf2d738ba5dc7644043d"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.803359 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca0e0c169040479ba95437ca504e819e"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "85dbd7e42e09c31b656c01d53a7c115a"
server_key_iv: "4eab5dedf48ddf2d738ba5dc7644043d"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.804503 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:15:57.805115 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.805285 20289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:15:57.805364 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal
uuid: "ca0e0c169040479ba95437ca504e819e"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "85dbd7e42e09c31b656c01d53a7c115a"
server_key_iv: "4eab5dedf48ddf2d738ba5dc7644043d"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.805423 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.816505 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.816802 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.816929 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.817126 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:57.817504 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:57.817548 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.817579 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:57.817595 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.822978 20289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46801
I20260928 23:15:57.823025 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46801 every 8 connection(s)
I20260928 23:15:57.823328 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/data/info.pb
I20260928 23:15:57.827256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20289
I20260928 23:15:57.827376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal/instance
I20260928 23:15:57.827548 392 external_mini_cluster.cc:1428] Setting key aff1fdce0423e9314f462bff10563b70
I20260928 23:15:57.828094 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.823951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39285 (local address 127.0.98.62:40339)
0928 23:15:57.823999 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.824000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.824217 (+ 217us) server_negotiation.cc:407] Connection header received
0928 23:15:57.824674 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.824675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.824683 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.824740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.825402 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.825889 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.826802 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.826978 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.827019 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.827838 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.827840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.827840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:57.827845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.827860 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:57.827864 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.827910 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.827988 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.828017 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:15:57.828289 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.823931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40339 (local address 127.0.98.2:39285)
0928 23:15:57.824102 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.824141 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.824603 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.824791 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.824794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.824802 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:57.825334 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.825339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.825959 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.825960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.826742 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.826744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.827028 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.827029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.827169 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.827675 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:57.827676 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:57.827730 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:57.827757 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:57.827915 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:57.827920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:57.827923 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.827990 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.828086 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 23:15:57.828424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--builtin_ntp_servers=127.0.98.20:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 23:15:57.828683 20417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40339
I20260928 23:15:57.828763 20417 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:57.828924 20417 heartbeater.cc:511] Master 127.0.98.62:40339 requested a full tablet report, sending...
I20260928 23:15:57.829276 20098 ts_manager.cc:194] Registered new tserver with Master: ca0e0c169040479ba95437ca504e819e (127.0.98.2:46801)
I20260928 23:15:57.829703 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39285
W20260928 23:15:57.907052 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:57.907249 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:57.907269 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:57.908449 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:57.908489 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:57.908516 20420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 23:15:57.909930 20420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:57.910135 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:57.910430 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:57.915396 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.915429 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:57.915426 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:57.915647 20420 server_base.cc:1034] running on GCE node
I20260928 23:15:57.915853 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:57.916075 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:57.917279 20420 hybrid_clock.cc:648] HybridClock initialized: now 1790637357917262 us; error 28 us; skew 500 ppm
I20260928 23:15:57.918498 20420 webserver.cc:460] Webserver started at http://127.0.98.3:43611/ using document root <none> and password file <none>
I20260928 23:15:57.918716 20420 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:57.918766 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:57.918892 20420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:57.919921 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data/instance:
uuid: "8637704a6da94667b4435bb10ca18906"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "6147e64daeeb0052f98c3d386e96777b"
server_key_iv: "efae0aac1649a6abcf20fb43e31cde7c"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.920249 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8637704a6da94667b4435bb10ca18906"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "6147e64daeeb0052f98c3d386e96777b"
server_key_iv: "efae0aac1649a6abcf20fb43e31cde7c"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.921468 20420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:15:57.922096 20434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.922261 20420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.922343 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal
uuid: "8637704a6da94667b4435bb10ca18906"
format_stamp: "Formatted at 2026-09-28 23:15:57 on dist-test-slave-6vgd"
server_key: "6147e64daeeb0052f98c3d386e96777b"
server_key_iv: "efae0aac1649a6abcf20fb43e31cde7c"
server_key_version: "encryptionkey@0"
I20260928 23:15:57.922401 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:57.931370 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:57.931641 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:57.931769 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:57.931953 20420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 23:15:57.932235 20420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 23:15:57.932265 20420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.932292 20420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 23:15:57.932317 20420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:57.937615 20420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37031
I20260928 23:15:57.937660 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37031 every 8 connection(s)
I20260928 23:15:57.937966 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/data/info.pb
I20260928 23:15:57.942688 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.938622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56035 (local address 127.0.98.62:40339)
0928 23:15:57.938699 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.938701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.938877 (+ 176us) server_negotiation.cc:407] Connection header received
0928 23:15:57.939353 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.939354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.939364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.939413 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.940035 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.940526 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.941552 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.941734 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.941782 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.942504 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.942504 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.942505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:57.942508 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.942518 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:57.942521 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.942554 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.942610 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.942630 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 23:15:57.942883 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.938601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40339 (local address 127.0.98.3:56035)
0928 23:15:57.938765 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.938797 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.939273 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.939470 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.939473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.939482 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:15:57.939955 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.939959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.940616 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.940617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.941481 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.941483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.941792 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.941793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.941906 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.942383 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 23:15:57.942384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:15:57.942412 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 23:15:57.942433 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:15:57.942556 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:15:57.942558 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:15:57.942560 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.942605 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.942678 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260928 23:15:57.942940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20420
I20260928 23:15:57.943028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal/instance
I20260928 23:15:57.943204 392 external_mini_cluster.cc:1428] Setting key 4b6dcc6784c12a78d3a6171244bc5d51
I20260928 23:15:57.943320 20548 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40339
I20260928 23:15:57.943401 20548 heartbeater.cc:464] Registering TS with master...
I20260928 23:15:57.943583 20548 heartbeater.cc:511] Master 127.0.98.62:40339 requested a full tablet report, sending...
I20260928 23:15:57.943941 20097 ts_manager.cc:194] Registered new tserver with Master: 8637704a6da94667b4435bb10ca18906 (127.0.98.3:37031)
I20260928 23:15:57.944288 20097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56035
I20260928 23:15:57.954790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 23:15:57.958781 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.956055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.0.98.62:40339)
0928 23:15:57.956113 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.956116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.956240 (+ 124us) server_negotiation.cc:407] Connection header received
0928 23:15:57.956263 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.956264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.956276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.956324 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:57.956479 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.956954 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.957725 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.957908 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.957961 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.958516 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:57.958517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:57.958518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:15:57.958522 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:57.958536 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:15:57.958539 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:57.958583 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.958646 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.958668 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 23:15:57.960188 20097 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 23:15:57.961735 20097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 23:15:57.967432 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38605 (local address 127.0.0.1:53832)
0928 23:15:57.963876 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.963885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.963948 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.964241 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.964242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.964244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.964304 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.964305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965246 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.965247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.966433 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.966436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967148 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.967221 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967233 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.967301 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.967360 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":76,"threads_started":1}
I20260928 23:15:57.967442 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.1:38605)
0928 23:15:57.964112 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.964116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.964125 (+ 9us) server_negotiation.cc:407] Connection header received
0928 23:15:57.964160 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.964162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.964175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.964243 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.964363 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965143 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.966525 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967075 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967136 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967228 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.967319 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.967360 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":87,"threads_started":1}
I20260928 23:15:57.967532 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.0.1:56912)
0928 23:15:57.963960 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.963968 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.964003 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.964364 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.964365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.964367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.964398 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.964399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965598 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.965599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.966388 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.966391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967316 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.967373 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967378 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.967420 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.967460 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260928 23:15:57.967725 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37031 (local address 127.0.0.1:49810)
0928 23:15:57.963840 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:57.963880 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:57.963946 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:57.964174 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:57.964177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:57.964180 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:57.964242 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.964244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965551 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.965552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.966610 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:57.966612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967525 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:57.967577 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967580 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 23:15:57.967636 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:15:57.967667 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":37,"thread_start_us":64,"threads_started":1}
I20260928 23:15:57.967739 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.0.98.2:46801)
0928 23:15:57.964145 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.964149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.964159 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:57.964190 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.964192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.964231 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.964301 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.964467 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965545 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.966469 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967134 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967199 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967282 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.967614 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.967653 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260928 23:15:57.967795 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:57.963657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.0.98.3:37031)
0928 23:15:57.963822 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:57.963826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:57.964021 (+ 195us) server_negotiation.cc:407] Connection header received
0928 23:15:57.964064 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:57.964066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:57.964080 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:57.964148 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:57.964324 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.965472 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.966686 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:57.967363 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:57.967422 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:57.967494 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:57.967668 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 23:15:57.967713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 23:15:57.968371 20482 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 (DEFAULT_TABLE table=test-table [id=1c8e2baaff704dcf9b5b3e6a7b817fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.968369 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 (DEFAULT_TABLE table=test-table [id=1c8e2baaff704dcf9b5b3e6a7b817fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.968523 20220 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 (DEFAULT_TABLE table=test-table [id=1c8e2baaff704dcf9b5b3e6a7b817fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 23:15:57.968710 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.968719 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.968873 20220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:57.972087 20564 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Bootstrap starting.
I20260928 23:15:57.972266 20562 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: Bootstrap starting.
I20260928 23:15:57.972687 20564 tablet_bootstrap.cc:654] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.972883 20562 tablet_bootstrap.cc:654] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.972980 20564 log.cc:824] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.973186 20562 log.cc:824] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.973318 20563 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Bootstrap starting.
I20260928 23:15:57.973882 20563 tablet_bootstrap.cc:654] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:57.973927 20562 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: No bootstrap required, opened a new log
I20260928 23:15:57.973982 20562 ts_tablet_manager.cc:1397] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 23:15:57.974176 20563 log.cc:824] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:57.974690 20564 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: No bootstrap required, opened a new log
I20260928 23:15:57.974761 20564 ts_tablet_manager.cc:1397] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 23:15:57.974784 20563 tablet_bootstrap.cc:492] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: No bootstrap required, opened a new log
I20260928 23:15:57.974841 20563 ts_tablet_manager.cc:1397] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 23:15:57.976426 20562 raft_consensus.cc:348] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.976593 20562 raft_consensus.cc:374] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.976625 20562 raft_consensus.cc:729] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aacc6dee2c144f98a45533e25cf93d3, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.976722 20563 raft_consensus.cc:348] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.976794 20562 consensus_queue.cc:260] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.976876 20563 raft_consensus.cc:374] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.976953 20563 raft_consensus.cc:729] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8637704a6da94667b4435bb10ca18906, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.977075 20563 consensus_queue.cc:260] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.977123 20562 ts_tablet_manager.cc:1428] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:15:57.977262 20286 heartbeater.cc:503] Master 127.0.98.62:40339 was elected leader, sending a full tablet report...
I20260928 23:15:57.977082 20564 raft_consensus.cc:348] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.977309 20564 raft_consensus.cc:374] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:57.977344 20563 ts_tablet_manager.cc:1428] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 23:15:57.977351 20564 raft_consensus.cc:729] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0e0c169040479ba95437ca504e819e, State: Initialized, Role: FOLLOWER
I20260928 23:15:57.977458 20548 heartbeater.cc:503] Master 127.0.98.62:40339 was elected leader, sending a full tablet report...
I20260928 23:15:57.977454 20564 consensus_queue.cc:260] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:57.977674 20564 ts_tablet_manager.cc:1428] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 23:15:57.977771 20417 heartbeater.cc:503] Master 127.0.98.62:40339 was elected leader, sending a full tablet report...
W20260928 23:15:58.073838 20418 tablet.cc:2369] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 23:15:58.074049 20569 raft_consensus.cc:482] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 23:15:58.074163 20569 raft_consensus.cc:504] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:58.074520 20569 leader_election.cc:288] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aacc6dee2c144f98a45533e25cf93d3 (127.0.98.1:38605), ca0e0c169040479ba95437ca504e819e (127.0.98.2:46801)
I20260928 23:15:58.077342 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.074864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41947 (local address 127.0.98.2:46801)
0928 23:15:58.074906 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:58.074908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:58.075024 (+ 116us) server_negotiation.cc:407] Connection header received
0928 23:15:58.075055 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:58.075056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:58.075065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:58.075115 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:58.075244 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.075704 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.076644 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.077016 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.077087 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:58.077097 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:58.077242 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 23:15:58.077277 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:15:58.077358 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.074695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38605 (local address 127.0.98.3:53125)
0928 23:15:58.074763 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:58.074777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:58.074843 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:58.074974 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:58.074975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:58.074977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:58.075050 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:58.075052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.075717 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.075718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:58.076574 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:58.076577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.077090 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.077091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:58.077160 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:58.077164 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:58.077229 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 23:15:58.077269 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":12}
I20260928 23:15:58.077348 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.074782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53125 (local address 127.0.98.1:38605)
0928 23:15:58.074826 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:58.074828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:58.074838 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:15:58.074899 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:58.074899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:58.074911 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:58.074963 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 23:15:58.075129 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.075643 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.076654 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.077015 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.077084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:58.077094 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:58.077244 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:15:58.077284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:15:58.077379 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.074808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46801 (local address 127.0.98.3:41947)
0928 23:15:58.074962 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:58.074970 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:15:58.075014 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:15:58.075123 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:15:58.075123 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:15:58.075124 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 23:15:58.075186 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:58.075187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.075782 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.075782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:58.076566 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:15:58.076569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.077097 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.077098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:15:58.077166 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:58.077170 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 23:15:58.077237 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 23:15:58.077275 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260928 23:15:58.077682 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba96ac3a3f1439ebedaf99dd31ad8e7" candidate_uuid: "8637704a6da94667b4435bb10ca18906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca0e0c169040479ba95437ca504e819e" is_pre_election: true
I20260928 23:15:58.077718 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba96ac3a3f1439ebedaf99dd31ad8e7" candidate_uuid: "8637704a6da94667b4435bb10ca18906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aacc6dee2c144f98a45533e25cf93d3" is_pre_election: true
I20260928 23:15:58.077836 20240 raft_consensus.cc:2454] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8637704a6da94667b4435bb10ca18906 in term 0.
I20260928 23:15:58.077836 20371 raft_consensus.cc:2454] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8637704a6da94667b4435bb10ca18906 in term 0.
I20260928 23:15:58.078042 20438 leader_election.cc:302] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aacc6dee2c144f98a45533e25cf93d3, 8637704a6da94667b4435bb10ca18906; no voters:
I20260928 23:15:58.078154 20569 raft_consensus.cc:2781] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 23:15:58.078207 20569 raft_consensus.cc:482] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 23:15:58.078238 20569 raft_consensus.cc:3037] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:58.078928 20569 raft_consensus.cc:504] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:58.079031 20569 leader_election.cc:288] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [CANDIDATE]: Term 1 election: Requested vote from peers 3aacc6dee2c144f98a45533e25cf93d3 (127.0.98.1:38605), ca0e0c169040479ba95437ca504e819e (127.0.98.2:46801)
I20260928 23:15:58.079195 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba96ac3a3f1439ebedaf99dd31ad8e7" candidate_uuid: "8637704a6da94667b4435bb10ca18906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aacc6dee2c144f98a45533e25cf93d3"
I20260928 23:15:58.079223 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba96ac3a3f1439ebedaf99dd31ad8e7" candidate_uuid: "8637704a6da94667b4435bb10ca18906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca0e0c169040479ba95437ca504e819e"
I20260928 23:15:58.079268 20240 raft_consensus.cc:3037] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:58.079324 20371 raft_consensus.cc:3037] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:58.079968 20240 raft_consensus.cc:2454] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8637704a6da94667b4435bb10ca18906 in term 1.
I20260928 23:15:58.080003 20371 raft_consensus.cc:2454] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8637704a6da94667b4435bb10ca18906 in term 1.
I20260928 23:15:58.080137 20438 leader_election.cc:302] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aacc6dee2c144f98a45533e25cf93d3, 8637704a6da94667b4435bb10ca18906; no voters:
I20260928 23:15:58.080250 20569 raft_consensus.cc:2781] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:58.080395 20569 raft_consensus.cc:686] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 1 LEADER]: Becoming Leader. State: Replica: 8637704a6da94667b4435bb10ca18906, State: Running, Role: LEADER
I20260928 23:15:58.080492 20569 consensus_queue.cc:237] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } }
I20260928 23:15:58.081326 20098 catalog_manager.cc:5495] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8637704a6da94667b4435bb10ca18906 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8637704a6da94667b4435bb10ca18906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8637704a6da94667b4435bb10ca18906" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20260928 23:15:58.113067 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.110918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.0.98.3:37031)
0928 23:15:58.110967 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:58.110970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:58.110976 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:15:58.111011 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:58.111012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:58.111029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:58.111084 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:15:58.111209 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.111658 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.112359 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:58.112541 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:58.112592 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:58.112761 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:15:58.112874 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:15:58.112933 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:58.112948 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 23:15:58.112965 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 23:15:58.114840 20371 raft_consensus.cc:1261] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 1 FOLLOWER]: Refusing update from remote peer 8637704a6da94667b4435bb10ca18906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:58.114840 20240 raft_consensus.cc:1261] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 1 FOLLOWER]: Refusing update from remote peer 8637704a6da94667b4435bb10ca18906: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:15:58.115192 20569 consensus_queue.cc:1035] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca0e0c169040479ba95437ca504e819e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:58.115347 20572 consensus_queue.cc:1035] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aacc6dee2c144f98a45533e25cf93d3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:15:58.117799 20575 mvcc.cc:205] Tried to move back new op lower bound from 7334450618836307968 to 7334450618698711040. Current Snapshot: MvccSnapshot[applied={T|T < 7334450618836307968}]
I20260928 23:15:58.118026 20577 mvcc.cc:205] Tried to move back new op lower bound from 7334450618836307968 to 7334450618698711040. Current Snapshot: MvccSnapshot[applied={T|T < 7334450618836307968}]
I20260928 23:15:58.118922 20576 mvcc.cc:205] Tried to move back new op lower bound from 7334450618836307968 to 7334450618698711040. Current Snapshot: MvccSnapshot[applied={T|T < 7334450618836307968}]
I20260928 23:15:58.119984 20098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:58.120065 20098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 23:15:58.120747 20098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1857e696e0c64efbb170dcb51788e9c4: Sending DeleteTablet for 3 replicas of tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7
I20260928 23:15:58.121021 20482 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:58 UTC) from {username='slave'} at 127.0.0.1:49810
I20260928 23:15:58.121102 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:58 UTC) from {username='slave'} at 127.0.0.1:56912
I20260928 23:15:58.121111 20220 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 23:15:58 UTC) from {username='slave'} at 127.0.0.1:53832
I20260928 23:15:58.121325 20586 tablet_replica.cc:330] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3: stopping tablet replica
I20260928 23:15:58.121416 20586 raft_consensus.cc:2229] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:58.121505 20586 raft_consensus.cc:2258] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 3aacc6dee2c144f98a45533e25cf93d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:58.121493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20158
I20260928 23:15:58.122113 20587 tablet_replica.cc:330] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: stopping tablet replica
I20260928 23:15:58.122185 20585 tablet_replica.cc:330] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: stopping tablet replica
I20260928 23:15:58.122195 20587 raft_consensus.cc:2229] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 23:15:58.122257 20585 raft_consensus.cc:2229] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 1 LEADER]: Raft consensus shutting down.
I20260928 23:15:58.122290 20587 raft_consensus.cc:2258] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:58.122331 20585 raft_consensus.cc:2258] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 23:15:58.122774 20585 ts_tablet_manager.cc:1905] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:58.123147 20587 ts_tablet_manager.cc:1905] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 23:15:58.124400 20585 ts_tablet_manager.cc:1918] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:58.124475 20585 log.cc:1196] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-2/wal/wals/0ba96ac3a3f1439ebedaf99dd31ad8e7
I20260928 23:15:58.124596 20587 ts_tablet_manager.cc:1918] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 23:15:58.124648 20587 log.cc:1196] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Deleting WAL directory at /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790637099107577-392-0/minicluster-data/ts-1/wal/wals/0ba96ac3a3f1439ebedaf99dd31ad8e7
I20260928 23:15:58.124747 20585 ts_tablet_manager.cc:1939] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P 8637704a6da94667b4435bb10ca18906: Deleting consensus metadata
I20260928 23:15:58.124871 20587 ts_tablet_manager.cc:1939] T 0ba96ac3a3f1439ebedaf99dd31ad8e7 P ca0e0c169040479ba95437ca504e819e: Deleting consensus metadata
I20260928 23:15:58.125111 20084 catalog_manager.cc:4841] TS 8637704a6da94667b4435bb10ca18906 (127.0.98.3:37031): tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 (table test-table [id=1c8e2baaff704dcf9b5b3e6a7b817fd7]) successfully deleted
I20260928 23:15:58.125316 20085 catalog_manager.cc:4841] TS ca0e0c169040479ba95437ca504e819e (127.0.98.2:46801): tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7 (table test-table [id=1c8e2baaff704dcf9b5b3e6a7b817fd7]) successfully deleted
W20260928 23:15:58.127014 20085 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38605 (error 108)
I20260928 23:15:58.127234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20289
I20260928 23:15:58.127791 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:58.127304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38605 (local address 127.0.0.1:53834)
0928 23:15:58.127711 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 23:15:58.127759 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260928 23:15:58.127911 20085 catalog_manager.cc:4584] TS 3aacc6dee2c144f98a45533e25cf93d3 (127.0.98.1:38605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ba96ac3a3f1439ebedaf99dd31ad8e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38605: connect: Connection refused (error 111)
I20260928 23:15:58.132021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20420
I20260928 23:15:58.136047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20067
2026-09-28T23:15:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (764 ms)
[----------] 6 tests from EncryptionPolicyTest (7832 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:15:58 dist-test-slave-6vgd krb5kdc[20591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:15:58 dist-test-slave-6vgd krb5kdc[20591](info): set up 2 sockets
Sep 28 23:15:58 dist-test-slave-6vgd krb5kdc[20591](info): commencing operation
krb5kdc: starting...
W20260928 23:15:59.080699 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:15:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:15:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:15:59.185935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40529
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:15:59.261986 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:15:59.262174 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:15:59.262204 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:15:59.263284 20604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:15:59.263324 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:15:59.263343 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:15:59.263357 20604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:15:59.263373 20604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:15:59.264768 20604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:15:59.264992 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:15:59.265182 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:15:59.269564 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:15:59.269541 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:59.269693 20604 server_base.cc:1034] running on GCE node
W20260928 23:15:59.269547 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:15:59.269936 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:15:59.270193 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:15:59.271323 20604 hybrid_clock.cc:648] HybridClock initialized: now 1790637359271312 us; error 29 us; skew 500 ppm
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:15:59.278760 20604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:15:59.279171 20604 webserver.cc:460] Webserver started at http://127.0.0.1:33233/ using document root <none> and password file <none>
I20260928 23:15:59.279369 20604 fs_manager.cc:359] Metadata directory not provided
I20260928 23:15:59.279414 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:15:59.279533 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:15:59.280515 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "f297ad3184b943eea4b8511af47b60c5"
format_stamp: "Formatted at 2026-09-28 23:15:59 on dist-test-slave-6vgd"
server_key: "596fcb581a9564bb244d7cfac09ea183"
server_key_iv: "cf366ae532a0027189134096b6f2f860"
server_key_version: "encryptionkey@0"
I20260928 23:15:59.280834 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "f297ad3184b943eea4b8511af47b60c5"
format_stamp: "Formatted at 2026-09-28 23:15:59 on dist-test-slave-6vgd"
server_key: "596fcb581a9564bb244d7cfac09ea183"
server_key_iv: "cf366ae532a0027189134096b6f2f860"
server_key_version: "encryptionkey@0"
I20260928 23:15:59.282045 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:15:59.282724 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:15:59.282882 20604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:15:59.282958 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "f297ad3184b943eea4b8511af47b60c5"
format_stamp: "Formatted at 2026-09-28 23:15:59 on dist-test-slave-6vgd"
server_key: "596fcb581a9564bb244d7cfac09ea183"
server_key_iv: "cf366ae532a0027189134096b6f2f860"
server_key_version: "encryptionkey@0"
I20260928 23:15:59.283018 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:15:59.294154 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:15:59.295759 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:15:59.295877 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:15:59.299537 20604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40529
I20260928 23:15:59.299583 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40529 every 8 connection(s)
I20260928 23:15:59.299870 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:15:59.300609 20672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:15:59.302780 20672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5: Bootstrap starting.
I20260928 23:15:59.303316 20672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5: Neither blocks nor log segments found. Creating new log.
I20260928 23:15:59.303571 20672 log.cc:824] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5: Log is configured to *not* fsync() on all Append() calls
I20260928 23:15:59.304188 20672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5: No bootstrap required, opened a new log
I20260928 23:15:59.305768 20672 raft_consensus.cc:348] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260928 23:15:59.305899 20672 raft_consensus.cc:374] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:15:59.305922 20672 raft_consensus.cc:729] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f297ad3184b943eea4b8511af47b60c5, State: Initialized, Role: FOLLOWER
I20260928 23:15:59.306070 20672 consensus_queue.cc:260] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260928 23:15:59.306129 20672 raft_consensus.cc:388] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:15:59.306160 20672 raft_consensus.cc:482] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:15:59.306205 20672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:15:59.306680 20672 raft_consensus.cc:504] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260928 23:15:59.306766 20672 leader_election.cc:302] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f297ad3184b943eea4b8511af47b60c5; no voters:
I20260928 23:15:59.306914 20672 leader_election.cc:288] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:15:59.306951 20675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:15:59.307096 20675 raft_consensus.cc:686] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [term 1 LEADER]: Becoming Leader. State: Replica: f297ad3184b943eea4b8511af47b60c5, State: Running, Role: LEADER
I20260928 23:15:59.307133 20672 sys_catalog.cc:564] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:15:59.307192 20675 consensus_queue.cc:237] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } }
I20260928 23:15:59.307533 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f297ad3184b943eea4b8511af47b60c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } } }
I20260928 23:15:59.307547 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f297ad3184b943eea4b8511af47b60c5. Latest consensus state: current_term: 1 leader_uuid: "f297ad3184b943eea4b8511af47b60c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f297ad3184b943eea4b8511af47b60c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40529 } } }
I20260928 23:15:59.307705 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:59.307729 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5 [sys.catalog]: This master's current role is: LEADER
I20260928 23:15:59.308022 20683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:15:59.308506 20683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:15:59.309672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20604
I20260928 23:15:59.309803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:15:59.309963 20683 catalog_manager.cc:1347] Generated new cluster ID: 45794a32bc3842999935ee5229b2b734
I20260928 23:15:59.309993 392 external_mini_cluster.cc:1428] Setting key 7345e17230bf4e910e6756d0eab48ba9
I20260928 23:15:59.310010 20683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:15:59.320828 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:59.310429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47825 (local address 127.0.0.1:40529)
0928 23:15:59.310590 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:59.310600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:59.310692 (+ 92us) server_negotiation.cc:407] Connection header received
0928 23:15:59.310844 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:59.310850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:59.310899 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:59.311022 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:59.311736 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:59.313093 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:59.314154 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:59.314748 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:59.314835 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:59.317470 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:59.317474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:59.317474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:59.317485 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:59.319278 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:59.319658 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:59.319661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:59.319662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:59.319764 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:59.319962 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:59.319963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:59.319963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:59.320278 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:59.320424 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:59.320522 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:15:59.320618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260928 23:15:59.321627 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:15:59.323915 20683 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:15:59.324359 20683 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:15:59.329476 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:15:59.322953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34069 (local address 127.0.0.1:40529)
0928 23:15:59.322993 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:15:59.322994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:15:59.323083 (+ 89us) server_negotiation.cc:407] Connection header received
0928 23:15:59.323123 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:15:59.323123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:15:59.323133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:15:59.323187 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:15:59.323734 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:59.324474 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:59.325441 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:15:59.325753 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:15:59.325818 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:15:59.326989 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:15:59.326990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:15:59.326991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:15:59.326996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:15:59.328397 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:59.328752 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:59.328753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:59.328753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:59.328806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:15:59.328994 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:15:59.328995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:15:59.328995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:15:59.329108 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:15:59.329225 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 23:15:59.329334 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 23:15:59.329391 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:15:59.330852 20683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f297ad3184b943eea4b8511af47b60c5: Generated new TSK 0
I20260928 23:15:59.331000 20683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:15:59.336758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20604
W20260928 23:15:59.340309 20589 connection.cc:535] client connection to 127.0.0.1:40529 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40529): Connection reset by peer (error 104)
2026-09-28T23:15:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20703](info): set up 2 sockets
Sep 28 23:15:59 dist-test-slave-6vgd krb5kdc[20703](info): commencing operation
krb5kdc: starting...
W20260928 23:16:00.264859 392 mini_kdc.cc:121] Time spent starting KDC: real 0.912s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:00 dist-test-slave-6vgd krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:00.371372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:00.448367 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:00.448566 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:00.448596 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:00.449743 20716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:00.449782 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:00.449800 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:00.449815 20716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:00.449831 20716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:00.451226 20716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:00.451452 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:00.451646 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:00.456162 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:00.456202 20716 server_base.cc:1034] running on GCE node
W20260928 23:16:00.456156 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:00.456154 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:00.456530 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:00.456818 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:00.457950 20716 hybrid_clock.cc:648] HybridClock initialized: now 1790637360457936 us; error 26 us; skew 500 ppm
Sep 28 23:16:00 dist-test-slave-6vgd krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:00.460121 20716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:00.460492 20716 webserver.cc:460] Webserver started at http://127.0.0.1:37071/ using document root <none> and password file <none>
I20260928 23:16:00.460672 20716 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:00.460717 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:00.460839 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:00.461843 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "35520493ee6a4970929b34c38e01ee7c"
format_stamp: "Formatted at 2026-09-28 23:16:00 on dist-test-slave-6vgd"
server_key: "43c34321603bc9e89c56c81679b4e6fc"
server_key_iv: "6fc3c429b6e17f9f4be9d5860e1b46d2"
server_key_version: "encryptionkey@0"
I20260928 23:16:00.462163 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "35520493ee6a4970929b34c38e01ee7c"
format_stamp: "Formatted at 2026-09-28 23:16:00 on dist-test-slave-6vgd"
server_key: "43c34321603bc9e89c56c81679b4e6fc"
server_key_iv: "6fc3c429b6e17f9f4be9d5860e1b46d2"
server_key_version: "encryptionkey@0"
I20260928 23:16:00.463299 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:16:00.463887 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:00.464027 20716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:00.464116 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "35520493ee6a4970929b34c38e01ee7c"
format_stamp: "Formatted at 2026-09-28 23:16:00 on dist-test-slave-6vgd"
server_key: "43c34321603bc9e89c56c81679b4e6fc"
server_key_iv: "6fc3c429b6e17f9f4be9d5860e1b46d2"
server_key_version: "encryptionkey@0"
I20260928 23:16:00.464174 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:00.481992 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:00.483632 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:00.483762 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:00.487342 20716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39091
I20260928 23:16:00.487376 20783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39091 every 8 connection(s)
I20260928 23:16:00.487681 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:00.488461 20784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:00.490568 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c: Bootstrap starting.
I20260928 23:16:00.491060 20784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:00.491283 20784 log.cc:824] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:00.491846 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c: No bootstrap required, opened a new log
I20260928 23:16:00.493348 20784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260928 23:16:00.493479 20784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:00.493506 20784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35520493ee6a4970929b34c38e01ee7c, State: Initialized, Role: FOLLOWER
I20260928 23:16:00.493611 20784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260928 23:16:00.493659 20784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:00.493676 20784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:00.493705 20784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:00.494128 20784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260928 23:16:00.494207 20784 leader_election.cc:302] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35520493ee6a4970929b34c38e01ee7c; no voters:
I20260928 23:16:00.494316 20784 leader_election.cc:288] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:00.494359 20787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:00.494510 20784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:00.494514 20787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [term 1 LEADER]: Becoming Leader. State: Replica: 35520493ee6a4970929b34c38e01ee7c, State: Running, Role: LEADER
I20260928 23:16:00.494680 20787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } }
I20260928 23:16:00.495162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20716
I20260928 23:16:00.495291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:00.495275 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35520493ee6a4970929b34c38e01ee7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260928 23:16:00.495361 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:00.495273 20789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35520493ee6a4970929b34c38e01ee7c. Latest consensus state: current_term: 1 leader_uuid: "35520493ee6a4970929b34c38e01ee7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35520493ee6a4970929b34c38e01ee7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39091 } } }
I20260928 23:16:00.495424 20789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:00.495476 392 external_mini_cluster.cc:1428] Setting key 69e9690b4a11e3c2b67ce23c539eccd6
I20260928 23:16:00.495733 20796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:00.496356 20796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:00.497783 20796 catalog_manager.cc:1347] Generated new cluster ID: d56b6cfa529545cc950b0e56e072ad86
I20260928 23:16:00.497828 20796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:00.501363 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:00.496102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.0.0.1:39091)
0928 23:16:00.496241 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:00.496252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:00.496284 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:16:00.496398 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:00.496405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:00.496459 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:00.496832 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:00.496921 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:00.498276 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:00.499320 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:00.499934 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:00.500047 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:00.500689 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:00.500693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:00.500693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:00.500888 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:00.500930 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:00.500945 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:00.501033 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:00.501105 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:16:00.501132 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 23:16:00.502297 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:00.506973 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:00.503722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38815 (local address 127.0.0.1:39091)
0928 23:16:00.503758 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:00.503760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:00.503829 (+ 69us) server_negotiation.cc:407] Connection header received
0928 23:16:00.503879 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:00.503880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:00.503888 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:00.503949 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:00.504071 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:00.504793 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:00.505742 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:00.506072 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:00.506141 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:00.506746 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:00.506747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:00.506748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:00.506752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:00.506767 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:00.506771 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:00.506830 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:00.506887 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:16:00.506898 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 23:16:00.510697 20796 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:00.511382 20796 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:00.518075 20796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35520493ee6a4970929b34c38e01ee7c: Generated new TSK 0
I20260928 23:16:00.518239 20796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:00.524222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20716
W20260928 23:16:00.527906 20701 connection.cc:535] client connection to 127.0.0.1:39091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39091): Connection reset by peer (error 104)
2026-09-28T23:16:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1187 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:00 dist-test-slave-6vgd krb5kdc[20815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:00 dist-test-slave-6vgd krb5kdc[20815](info): set up 2 sockets
Sep 28 23:16:00 dist-test-slave-6vgd krb5kdc[20815](info): commencing operation
krb5kdc: starting...
W20260928 23:16:01.456943 392 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:01 dist-test-slave-6vgd krb5kdc[20815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:01.565127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:01.642040 20828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:01.642227 20828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:01.642248 20828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:01.643404 20828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:01.643448 20828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:01.643461 20828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:01.643472 20828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:01.643492 20828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:01.644942 20828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:01.645183 20828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:01.645421 20828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:01.649756 20836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:01.649770 20833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:01.649832 20834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:01.649951 20828 server_base.cc:1034] running on GCE node
I20260928 23:16:01.650159 20828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:01.650378 20828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:01.651537 20828 hybrid_clock.cc:648] HybridClock initialized: now 1790637361651518 us; error 28 us; skew 500 ppm
Sep 28 23:16:01 dist-test-slave-6vgd krb5kdc[20815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:01.653887 20828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:01.654295 20828 webserver.cc:460] Webserver started at http://127.0.0.1:43947/ using document root <none> and password file <none>
I20260928 23:16:01.654501 20828 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:01.654549 20828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:01.654668 20828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:01.655675 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "4d29aafba9c14570a45f9ef1f75d4c83"
format_stamp: "Formatted at 2026-09-28 23:16:01 on dist-test-slave-6vgd"
server_key: "809546898807cb4ae7c75ff405d5a0df"
server_key_iv: "05adfc118bf568d41dd48f9e278e0781"
server_key_version: "encryptionkey@0"
I20260928 23:16:01.656006 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d29aafba9c14570a45f9ef1f75d4c83"
format_stamp: "Formatted at 2026-09-28 23:16:01 on dist-test-slave-6vgd"
server_key: "809546898807cb4ae7c75ff405d5a0df"
server_key_iv: "05adfc118bf568d41dd48f9e278e0781"
server_key_version: "encryptionkey@0"
I20260928 23:16:01.657176 20828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:16:01.657811 20843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:01.657954 20828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:01.658032 20828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "4d29aafba9c14570a45f9ef1f75d4c83"
format_stamp: "Formatted at 2026-09-28 23:16:01 on dist-test-slave-6vgd"
server_key: "809546898807cb4ae7c75ff405d5a0df"
server_key_iv: "05adfc118bf568d41dd48f9e278e0781"
server_key_version: "encryptionkey@0"
I20260928 23:16:01.658092 20828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:01.661137 20828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:01.662741 20828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:01.662873 20828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:01.666476 20828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37671
I20260928 23:16:01.666502 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37671 every 8 connection(s)
I20260928 23:16:01.666788 20828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:01.667717 20896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:01.668915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20828
I20260928 23:16:01.669027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:01.669176 392 external_mini_cluster.cc:1428] Setting key aabf6ca3a22de160cded75de2fff8af5
I20260928 23:16:01.670050 20896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83: Bootstrap starting.
I20260928 23:16:01.670629 20896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:01.670928 20896 log.cc:824] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:01.671445 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:01.669636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46411 (local address 127.0.0.1:37671)
0928 23:16:01.669780 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:01.669787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:01.669809 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:16:01.670218 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:01.670225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:01.670250 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:01.670366 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:01.671042 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:01.671046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:01.671046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:01.671051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:01.671083 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:01.671093 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:01.671173 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:01.671231 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:16:01.671270 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260928 23:16:01.671648 20896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83: No bootstrap required, opened a new log
I20260928 23:16:01.672353 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:01.673640 20896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260928 23:16:01.673796 20896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:01.673831 20896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d29aafba9c14570a45f9ef1f75d4c83, State: Initialized, Role: FOLLOWER
I20260928 23:16:01.673944 20896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260928 23:16:01.674019 20896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:01.674057 20896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:01.674103 20896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:01.674589 20896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260928 23:16:01.674655 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:01.673621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58943 (local address 127.0.0.1:37671)
0928 23:16:01.673652 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:01.673653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:01.673753 (+ 100us) server_negotiation.cc:407] Connection header received
0928 23:16:01.673810 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:01.673811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:01.673816 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:01.673859 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:01.674477 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:01.674479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:01.674479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:01.674483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:01.674495 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:01.674499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:01.674538 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:01.674586 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:16:01.674597 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:16:01.674690 20896 leader_election.cc:302] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d29aafba9c14570a45f9ef1f75d4c83; no voters:
I20260928 23:16:01.674847 20896 leader_election.cc:288] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:01.675000 20907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:01.675084 20896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:01.675156 20907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [term 1 LEADER]: Becoming Leader. State: Replica: 4d29aafba9c14570a45f9ef1f75d4c83, State: Running, Role: LEADER
I20260928 23:16:01.675271 20907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260928 23:16:01.675571 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260928 23:16:01.675673 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:01.675850 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d29aafba9c14570a45f9ef1f75d4c83. Latest consensus state: current_term: 1 leader_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d29aafba9c14570a45f9ef1f75d4c83" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260928 23:16:01.675912 20915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:01.675971 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:01.676621 20915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:01.678081 20915 catalog_manager.cc:1347] Generated new cluster ID: 42d6ceccaf7d4e48a4340c546738a68b
I20260928 23:16:01.678135 20915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:01.704717 20915 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:01.704893 20915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:01.707399 20915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d29aafba9c14570a45f9ef1f75d4c83: Generated new TSK 0
I20260928 23:16:01.707566 20915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:01.710574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20828
2026-09-28T23:16:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1186 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:01 dist-test-slave-6vgd krb5kdc[20927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:01 dist-test-slave-6vgd krb5kdc[20927](info): set up 2 sockets
Sep 28 23:16:01 dist-test-slave-6vgd krb5kdc[20927](info): commencing operation
krb5kdc: starting...
W20260928 23:16:02.656894 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:02.764355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:02.841202 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:02.841416 20940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:02.841439 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:02.842548 20940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:02.842590 20940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:02.842603 20940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:02.842628 20940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:02.842650 20940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:02.844098 20940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:02.844318 20940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:02.844520 20940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:02.848848 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:02.848896 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:02.848867 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:02.849032 20940 server_base.cc:1034] running on GCE node
I20260928 23:16:02.849371 20940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:02.849641 20940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:02.850780 20940 hybrid_clock.cc:648] HybridClock initialized: now 1790637362850759 us; error 34 us; skew 500 ppm
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:02.852983 20940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:02.853418 20940 webserver.cc:460] Webserver started at http://127.0.0.1:45289/ using document root <none> and password file <none>
I20260928 23:16:02.853601 20940 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:02.853642 20940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:02.853751 20940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:02.854730 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "05d77a4a13314b71a804c3592f1e5ab0"
format_stamp: "Formatted at 2026-09-28 23:16:02 on dist-test-slave-6vgd"
server_key: "038fe87306e39b8fba1c6eda2151ba3a"
server_key_iv: "928a2ffc6e3bcfdae4f1cea81735c46b"
server_key_version: "encryptionkey@0"
I20260928 23:16:02.855065 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "05d77a4a13314b71a804c3592f1e5ab0"
format_stamp: "Formatted at 2026-09-28 23:16:02 on dist-test-slave-6vgd"
server_key: "038fe87306e39b8fba1c6eda2151ba3a"
server_key_iv: "928a2ffc6e3bcfdae4f1cea81735c46b"
server_key_version: "encryptionkey@0"
I20260928 23:16:02.856225 20940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:02.856843 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:02.856997 20940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:02.857077 20940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "05d77a4a13314b71a804c3592f1e5ab0"
format_stamp: "Formatted at 2026-09-28 23:16:02 on dist-test-slave-6vgd"
server_key: "038fe87306e39b8fba1c6eda2151ba3a"
server_key_iv: "928a2ffc6e3bcfdae4f1cea81735c46b"
server_key_version: "encryptionkey@0"
I20260928 23:16:02.857128 20940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:02.867827 20940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:02.869758 20940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:02.869863 20940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:02.873492 20940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39041
I20260928 23:16:02.873519 21007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39041 every 8 connection(s)
I20260928 23:16:02.873826 20940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:02.874631 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:02.876863 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0: Bootstrap starting.
I20260928 23:16:02.877418 21008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:02.877664 21008 log.cc:824] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:02.878132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 20940
I20260928 23:16:02.878254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:02.878376 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0: No bootstrap required, opened a new log
I20260928 23:16:02.878434 392 external_mini_cluster.cc:1428] Setting key 29a5c2592cc9b1a5903644f00b7b9010
I20260928 23:16:02.880535 21008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } }
I20260928 23:16:02.880671 21008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:02.880693 21008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d77a4a13314b71a804c3592f1e5ab0, State: Initialized, Role: FOLLOWER
I20260928 23:16:02.880805 21008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } }
I20260928 23:16:02.880867 21008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:02.880885 21008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:02.880930 21008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:02.881476 21008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } }
I20260928 23:16:02.881564 21008 leader_election.cc:302] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05d77a4a13314b71a804c3592f1e5ab0; no voters:
I20260928 23:16:02.881711 21008 leader_election.cc:288] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:02.881745 21013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:02.881893 21013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 05d77a4a13314b71a804c3592f1e5ab0, State: Running, Role: LEADER
I20260928 23:16:02.881968 21008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:02.881975 21013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } }
I20260928 23:16:02.882326 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05d77a4a13314b71a804c3592f1e5ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } } }
I20260928 23:16:02.882462 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:02.882633 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05d77a4a13314b71a804c3592f1e5ab0. Latest consensus state: current_term: 1 leader_uuid: "05d77a4a13314b71a804c3592f1e5ab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05d77a4a13314b71a804c3592f1e5ab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39041 } } }
I20260928 23:16:02.882733 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:02.883039 21022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:02.883656 21022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:02.885222 21022 catalog_manager.cc:1347] Generated new cluster ID: deea56b115f741239011d03abf87a740
I20260928 23:16:02.885272 21022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:16:02.889261 21022 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:02.889943 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:02.878894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35919 (local address 127.0.0.1:39041)
0928 23:16:02.879069 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:02.879079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:02.879103 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:16:02.879604 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:02.879610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:02.879656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:02.879763 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:02.880522 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:02.881820 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:02.882775 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:02.883323 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:02.883431 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:02.886148 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:02.886152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:02.886152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:02.886164 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:02.888115 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:02.888469 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:02.888472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:02.888472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:02.888580 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:02.888780 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:02.888782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:02.888782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:02.889044 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:02.889186 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:02.889631 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 23:16:02.889753 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260928 23:16:02.889951 21022 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:02.890818 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:02.893736 21022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05d77a4a13314b71a804c3592f1e5ab0: Generated new TSK 0
I20260928 23:16:02.893918 21022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:02.899296 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:02.893433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43429 (local address 127.0.0.1:39041)
0928 23:16:02.893465 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:02.893467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:02.893659 (+ 192us) server_negotiation.cc:407] Connection header received
0928 23:16:02.893714 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:02.893715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:02.893725 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:02.893781 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:02.894347 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:02.894817 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:02.895828 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:02.896023 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:02.896064 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:02.897227 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:02.897228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:02.897229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:02.897235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:02.898395 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:02.898732 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:02.898733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:02.898733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:02.898773 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:02.898922 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:02.898922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:02.898923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:02.899002 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:02.899061 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:02.899182 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 23:16:02.899232 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:16:02.901069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 20940
2026-09-28T23:16:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[21039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[21039](info): set up 2 sockets
Sep 28 23:16:02 dist-test-slave-6vgd krb5kdc[21039](info): commencing operation
krb5kdc: starting...
W20260928 23:16:03.844900 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:03 dist-test-slave-6vgd krb5kdc[21039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:03.953523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:04.031916 21052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:04.032104 21052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:04.032124 21052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:04.033188 21052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:04.033262 21052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:04.033280 21052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:04.033291 21052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:04.033311 21052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:04.034759 21052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:04.034943 21052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:04.035106 21052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:04.039428 21060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:04.039484 21057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:04.039494 21052 server_base.cc:1034] running on GCE node
W20260928 23:16:04.039585 21058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:04.039963 21052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:04.040217 21052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:04.041343 21052 hybrid_clock.cc:648] HybridClock initialized: now 1790637364041328 us; error 31 us; skew 500 ppm
Sep 28 23:16:04 dist-test-slave-6vgd krb5kdc[21039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:04.043517 21052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:04.043901 21052 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260928 23:16:04.044094 21052 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:04.044140 21052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:04.044267 21052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:04.045277 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "73c49521d7a54e7896b034bd2c26ade6"
format_stamp: "Formatted at 2026-09-28 23:16:04 on dist-test-slave-6vgd"
server_key: "ed640abade017ad0ead31ff49a442641"
server_key_iv: "fd595125ad172645acdac5a008b4a17b"
server_key_version: "encryptionkey@0"
I20260928 23:16:04.045608 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "73c49521d7a54e7896b034bd2c26ade6"
format_stamp: "Formatted at 2026-09-28 23:16:04 on dist-test-slave-6vgd"
server_key: "ed640abade017ad0ead31ff49a442641"
server_key_iv: "fd595125ad172645acdac5a008b4a17b"
server_key_version: "encryptionkey@0"
I20260928 23:16:04.046761 21052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:16:04.047410 21067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:04.047554 21052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:04.047628 21052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "73c49521d7a54e7896b034bd2c26ade6"
format_stamp: "Formatted at 2026-09-28 23:16:04 on dist-test-slave-6vgd"
server_key: "ed640abade017ad0ead31ff49a442641"
server_key_iv: "fd595125ad172645acdac5a008b4a17b"
server_key_version: "encryptionkey@0"
I20260928 23:16:04.047693 21052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:04.074910 21052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:04.076674 21052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:04.076807 21052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:04.080413 21052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40341
I20260928 23:16:04.080457 21119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40341 every 8 connection(s)
I20260928 23:16:04.080783 21052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:04.081661 21120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:04.083807 21120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6: Bootstrap starting.
I20260928 23:16:04.084350 21120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:04.084599 21120 log.cc:824] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:04.085266 21120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6: No bootstrap required, opened a new log
I20260928 23:16:04.086786 21120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } }
I20260928 23:16:04.086922 21120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:04.086943 21120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c49521d7a54e7896b034bd2c26ade6, State: Initialized, Role: FOLLOWER
I20260928 23:16:04.087085 21120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } }
I20260928 23:16:04.087167 21120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:04.087193 21120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:04.087239 21120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:04.087445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21052
I20260928 23:16:04.087558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:04.087724 392 external_mini_cluster.cc:1428] Setting key c74e2090f42b50fac0f935deb06e0c6b
I20260928 23:16:04.087762 21120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } }
I20260928 23:16:04.087863 21120 leader_election.cc:302] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73c49521d7a54e7896b034bd2c26ade6; no voters:
I20260928 23:16:04.088028 21120 leader_election.cc:288] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:04.088064 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:04.088550 21120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:04.088567 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [term 1 LEADER]: Becoming Leader. State: Replica: 73c49521d7a54e7896b034bd2c26ade6, State: Running, Role: LEADER
I20260928 23:16:04.088667 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } }
I20260928 23:16:04.089399 21127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73c49521d7a54e7896b034bd2c26ade6. Latest consensus state: current_term: 1 leader_uuid: "73c49521d7a54e7896b034bd2c26ade6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } } }
I20260928 23:16:04.089485 21127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:04.089533 21126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73c49521d7a54e7896b034bd2c26ade6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73c49521d7a54e7896b034bd2c26ade6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40341 } } }
I20260928 23:16:04.089620 21126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:04.089775 21134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:04.090291 21134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:04.092648 21134 catalog_manager.cc:1347] Generated new cluster ID: baacb01c0c6e4d2790db37d574e3d4e8
I20260928 23:16:04.092710 21134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:04.093142 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:04.088104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52493 (local address 127.0.0.1:40341)
0928 23:16:04.088261 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:04.088271 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:04.088307 (+ 36us) server_negotiation.cc:407] Connection header received
0928 23:16:04.088461 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:04.088469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:04.088555 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:04.088918 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:04.088989 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:04.090430 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:04.091301 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:04.091682 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:04.091750 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:04.092497 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:04.092501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:04.092502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:04.092711 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:04.092747 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:04.092761 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:04.092821 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:04.092893 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 23:16:04.092950 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260928 23:16:04.097615 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:04.102249 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:04.099009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44959 (local address 127.0.0.1:40341)
0928 23:16:04.099044 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:04.099046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:04.099159 (+ 113us) server_negotiation.cc:407] Connection header received
0928 23:16:04.099209 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:04.099210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:04.099221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:04.099284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:04.099398 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:04.100154 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:04.101053 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:04.101385 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:04.101451 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:04.102029 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:04.102031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:04.102031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:04.102036 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:04.102052 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:04.102056 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:04.102108 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:04.102151 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:16:04.102174 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:04.114244 21134 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:04.114733 21134 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:04.120211 21134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73c49521d7a54e7896b034bd2c26ade6: Generated new TSK 0
I20260928 23:16:04.120358 21134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:04.126093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21052
2026-09-28T23:16:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:04 dist-test-slave-6vgd krb5kdc[21151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:04 dist-test-slave-6vgd krb5kdc[21151](info): set up 2 sockets
Sep 28 23:16:04 dist-test-slave-6vgd krb5kdc[21151](info): commencing operation
krb5kdc: starting...
W20260928 23:16:05.056877 392 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:05 dist-test-slave-6vgd krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:05.160132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:05.235230 21164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:05.235424 21164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:05.235445 21164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:05.236493 21164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:05.236524 21164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:05.236537 21164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:05.236548 21164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:05.236562 21164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:05.237923 21164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:05.238103 21164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:05.238265 21164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:05.242602 21172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:05.242636 21169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:05.242626 21170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:05.242808 21164 server_base.cc:1034] running on GCE node
I20260928 23:16:05.243073 21164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:05.243350 21164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:05.244485 21164 hybrid_clock.cc:648] HybridClock initialized: now 1790637365244481 us; error 37 us; skew 500 ppm
Sep 28 23:16:05 dist-test-slave-6vgd krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:05.249967 21164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:05.250327 21164 webserver.cc:460] Webserver started at http://127.0.0.1:41217/ using document root <none> and password file <none>
I20260928 23:16:05.250484 21164 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:05.250520 21164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:05.250607 21164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:05.251529 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "f798982e19f547e380d25b09772a0594"
format_stamp: "Formatted at 2026-09-28 23:16:05 on dist-test-slave-6vgd"
server_key: "53be755923960e72db026f1f4122ea61"
server_key_iv: "435e0c03010461eb80200513bfe6bb29"
server_key_version: "encryptionkey@0"
I20260928 23:16:05.251807 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "f798982e19f547e380d25b09772a0594"
format_stamp: "Formatted at 2026-09-28 23:16:05 on dist-test-slave-6vgd"
server_key: "53be755923960e72db026f1f4122ea61"
server_key_iv: "435e0c03010461eb80200513bfe6bb29"
server_key_version: "encryptionkey@0"
I20260928 23:16:05.252900 21164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:16:05.253556 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:05.253711 21164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:05.253798 21164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "f798982e19f547e380d25b09772a0594"
format_stamp: "Formatted at 2026-09-28 23:16:05 on dist-test-slave-6vgd"
server_key: "53be755923960e72db026f1f4122ea61"
server_key_iv: "435e0c03010461eb80200513bfe6bb29"
server_key_version: "encryptionkey@0"
I20260928 23:16:05.253855 21164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:05.256867 21164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:05.258486 21164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:05.258603 21164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:05.262393 21164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37513
I20260928 23:16:05.262427 21231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37513 every 8 connection(s)
I20260928 23:16:05.262706 21164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:05.263285 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:05.263401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21164
I20260928 23:16:05.263512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:05.263670 392 external_mini_cluster.cc:1428] Setting key 79945f7309bc2458f12845356b08c04b
I20260928 23:16:05.265880 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:05.264112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34065 (local address 127.0.0.1:37513)
0928 23:16:05.264283 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:05.264292 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:05.264324 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:16:05.264751 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:05.264760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:05.264802 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:05.264947 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:05.265489 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:05.265493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:05.265493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:05.265497 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:05.265537 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:05.265550 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:05.265624 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:05.265658 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 23:16:05.265694 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260928 23:16:05.266386 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: Bootstrap starting.
I20260928 23:16:05.266727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:05.266976 21232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:05.267262 21232 log.cc:824] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:05.267939 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: No bootstrap required, opened a new log
I20260928 23:16:05.268963 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:05.268087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38107 (local address 127.0.0.1:37513)
0928 23:16:05.268131 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:05.268132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:05.268261 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:16:05.268326 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:05.268326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:05.268330 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:05.268371 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:05.268811 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:05.268812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:05.268813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:05.268816 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:05.268825 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:05.268829 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:05.268866 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:05.268916 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:16:05.268925 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:16:05.270097 21232 raft_consensus.cc:348] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } }
I20260928 23:16:05.270228 21232 raft_consensus.cc:374] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:05.270255 21232 raft_consensus.cc:729] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f798982e19f547e380d25b09772a0594, State: Initialized, Role: FOLLOWER
I20260928 23:16:05.270334 21232 consensus_queue.cc:260] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } }
I20260928 23:16:05.270390 21232 raft_consensus.cc:388] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:05.270409 21232 raft_consensus.cc:482] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:05.270438 21232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:05.270983 21232 raft_consensus.cc:504] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } }
I20260928 23:16:05.271068 21232 leader_election.cc:302] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f798982e19f547e380d25b09772a0594; no voters:
I20260928 23:16:05.271188 21232 leader_election.cc:288] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:05.271229 21242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:05.271400 21232 sys_catalog.cc:564] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:05.271399 21242 raft_consensus.cc:686] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [term 1 LEADER]: Becoming Leader. State: Replica: f798982e19f547e380d25b09772a0594, State: Running, Role: LEADER
I20260928 23:16:05.271564 21242 consensus_queue.cc:237] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } }
I20260928 23:16:05.271883 21243 sys_catalog.cc:455] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f798982e19f547e380d25b09772a0594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } } }
I20260928 23:16:05.271889 21244 sys_catalog.cc:455] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f798982e19f547e380d25b09772a0594. Latest consensus state: current_term: 1 leader_uuid: "f798982e19f547e380d25b09772a0594" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f798982e19f547e380d25b09772a0594" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37513 } } }
I20260928 23:16:05.272011 21243 sys_catalog.cc:458] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:05.272019 21244 sys_catalog.cc:458] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594 [sys.catalog]: This master's current role is: LEADER
W20260928 23:16:05.273000 21258 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:05.273041 21258 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:05.273077 21251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:05.273224 21251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:05.274464 21251 catalog_manager.cc:1347] Generated new cluster ID: ed689f820e404d82942b2e11a445bcd9
I20260928 23:16:05.274513 21251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:05.287897 21251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:05.288043 21251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:05.292171 21251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f798982e19f547e380d25b09772a0594: Generated new TSK 0
I20260928 23:16:05.292352 21251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:05.302634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21164
2026-09-28T23:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:05 dist-test-slave-6vgd krb5kdc[21263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:05 dist-test-slave-6vgd krb5kdc[21263](info): set up 2 sockets
Sep 28 23:16:05 dist-test-slave-6vgd krb5kdc[21263](info): commencing operation
krb5kdc: starting...
W20260928 23:16:06.237161 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:06.343097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:06.421343 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:06.421540 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:06.421571 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:06.422672 21276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:06.422712 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:06.422731 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:06.422747 21276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:06.422762 21276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:06.424180 21276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:06.424410 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:06.424619 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:06.428959 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:06.429085 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:06.428973 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:06.429198 21276 server_base.cc:1034] running on GCE node
I20260928 23:16:06.429430 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:06.429685 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:06.430812 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790637366430804 us; error 26 us; skew 500 ppm
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:06.433033 21276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:06.433472 21276 webserver.cc:460] Webserver started at http://127.0.0.1:42971/ using document root <none> and password file <none>
I20260928 23:16:06.433669 21276 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:06.433712 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:06.433816 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:06.434788 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "de2cadf112da4e299729a41afdd88645"
format_stamp: "Formatted at 2026-09-28 23:16:06 on dist-test-slave-6vgd"
server_key: "6d1166d2716ee6bbcccd8a93e71aa6c6"
server_key_iv: "fc415cc468cdaaf4e7910b1c835eb75e"
server_key_version: "encryptionkey@0"
I20260928 23:16:06.435097 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "de2cadf112da4e299729a41afdd88645"
format_stamp: "Formatted at 2026-09-28 23:16:06 on dist-test-slave-6vgd"
server_key: "6d1166d2716ee6bbcccd8a93e71aa6c6"
server_key_iv: "fc415cc468cdaaf4e7910b1c835eb75e"
server_key_version: "encryptionkey@0"
I20260928 23:16:06.436213 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:16:06.436790 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:06.436940 21276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:06.437021 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "de2cadf112da4e299729a41afdd88645"
format_stamp: "Formatted at 2026-09-28 23:16:06 on dist-test-slave-6vgd"
server_key: "6d1166d2716ee6bbcccd8a93e71aa6c6"
server_key_iv: "fc415cc468cdaaf4e7910b1c835eb75e"
server_key_version: "encryptionkey@0"
I20260928 23:16:06.437070 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:06.449060 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:06.451016 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:06.451120 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:06.454622 21276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33387
I20260928 23:16:06.454658 21343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33387 every 8 connection(s)
I20260928 23:16:06.454948 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:06.455693 21344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:06.456822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21276
I20260928 23:16:06.456913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:06.457068 392 external_mini_cluster.cc:1428] Setting key 473b4cf85b44cc91e6e7a0b9cd308cec
I20260928 23:16:06.458254 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645: Bootstrap starting.
I20260928 23:16:06.458854 21344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:06.459133 21344 log.cc:824] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:06.459827 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645: No bootstrap required, opened a new log
I20260928 23:16:06.461889 21344 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260928 23:16:06.462062 21344 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:06.462105 21344 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2cadf112da4e299729a41afdd88645, State: Initialized, Role: FOLLOWER
I20260928 23:16:06.462229 21344 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260928 23:16:06.462303 21344 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:06.462324 21344 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:06.462378 21344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:06.462867 21344 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260928 23:16:06.462946 21344 leader_election.cc:302] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2cadf112da4e299729a41afdd88645; no voters:
I20260928 23:16:06.463101 21344 leader_election.cc:288] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:06.463135 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:06.463240 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [term 1 LEADER]: Becoming Leader. State: Replica: de2cadf112da4e299729a41afdd88645, State: Running, Role: LEADER
I20260928 23:16:06.463315 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } }
I20260928 23:16:06.463349 21344 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:16:06.463600 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2cadf112da4e299729a41afdd88645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260928 23:16:06.463697 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:06.463814 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2cadf112da4e299729a41afdd88645. Latest consensus state: current_term: 1 leader_uuid: "de2cadf112da4e299729a41afdd88645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2cadf112da4e299729a41afdd88645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33387 } } }
I20260928 23:16:06.463869 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:06.463969 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:06.464723 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:06.467084 21354 catalog_manager.cc:1347] Generated new cluster ID: 0764fbc5f1c34a639805b6acc6854cac
I20260928 23:16:06.467135 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:06.467631 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:06.457530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:58969 (local address 127.0.0.1:33387)
0928 23:16:06.457773 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:06.457783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:06.457813 (+ 30us) server_negotiation.cc:407] Connection header received
0928 23:16:06.458314 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:06.458323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:06.458384 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:06.458524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:06.459081 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:06.460359 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:06.461074 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:06.461719 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:06.461824 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:06.464089 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:06.464093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:06.464094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:06.464106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:06.465904 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:06.466300 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:06.466302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:06.466303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:06.466414 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:06.466654 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:06.466655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:06.466656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:06.466940 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:06.467083 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:06.467329 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 23:16:06.467432 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":69,"threads_started":1}
I20260928 23:16:06.468441 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:06.479945 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:06.469997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:33489 (local address 127.0.0.1:33387)
0928 23:16:06.470041 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:06.470043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:06.473357 (+ 3314us) server_negotiation.cc:407] Connection header received
0928 23:16:06.473439 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:06.473440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:06.473449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:06.473474 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:06.474079 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:06.474849 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:06.475784 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:06.476128 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:06.476194 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:06.477446 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:06.477448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:06.477448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:06.477455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:06.478890 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:06.479247 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:06.479248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:06.479248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:06.479303 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:06.479478 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:06.479479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:06.479479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:06.479595 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:06.479702 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:06.479790 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 23:16:06.479851 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:16:06.481405 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:06.481858 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:06.490235 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2cadf112da4e299729a41afdd88645: Generated new TSK 0
I20260928 23:16:06.490401 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:06.497677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21276
W20260928 23:16:06.501605 21261 connection.cc:535] client connection to 127.0.0.1:33387 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33387): Connection reset by peer (error 104)
2026-09-28T23:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21375](info): set up 2 sockets
Sep 28 23:16:06 dist-test-slave-6vgd krb5kdc[21375](info): commencing operation
krb5kdc: starting...
W20260928 23:16:07.436905 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:07 dist-test-slave-6vgd krb5kdc[21375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:07.542240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:07.621320 21388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:07.621572 21388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:07.621603 21388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:07.622813 21388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:07.622859 21388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:07.622879 21388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:07.622895 21388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:07.622910 21388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:07.624408 21388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:07.624647 21388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:07.624857 21388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:07.629434 21394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:07.629441 21396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:07.629441 21393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:07.629638 21388 server_base.cc:1034] running on GCE node
I20260928 23:16:07.629931 21388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:07.630234 21388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:07.631371 21388 hybrid_clock.cc:648] HybridClock initialized: now 1790637367631362 us; error 29 us; skew 500 ppm
Sep 28 23:16:07 dist-test-slave-6vgd krb5kdc[21375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:07.633730 21388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:07.634150 21388 webserver.cc:460] Webserver started at http://127.0.0.1:44505/ using document root <none> and password file <none>
I20260928 23:16:07.634325 21388 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:07.634361 21388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:07.634451 21388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:07.635458 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "cbe2ca5c36ec409d83fbac9f4948684f"
format_stamp: "Formatted at 2026-09-28 23:16:07 on dist-test-slave-6vgd"
server_key: "df7947c7ab5e4b19a771912c965fa453"
server_key_iv: "3214679f8dc8c75da856a9df3775f118"
server_key_version: "encryptionkey@0"
I20260928 23:16:07.635787 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbe2ca5c36ec409d83fbac9f4948684f"
format_stamp: "Formatted at 2026-09-28 23:16:07 on dist-test-slave-6vgd"
server_key: "df7947c7ab5e4b19a771912c965fa453"
server_key_iv: "3214679f8dc8c75da856a9df3775f118"
server_key_version: "encryptionkey@0"
I20260928 23:16:07.636968 21388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 23:16:07.637692 21403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:07.637845 21388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:07.637926 21388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "cbe2ca5c36ec409d83fbac9f4948684f"
format_stamp: "Formatted at 2026-09-28 23:16:07 on dist-test-slave-6vgd"
server_key: "df7947c7ab5e4b19a771912c965fa453"
server_key_iv: "3214679f8dc8c75da856a9df3775f118"
server_key_version: "encryptionkey@0"
I20260928 23:16:07.637981 21388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:07.652791 21388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:07.654556 21388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:07.654670 21388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:07.658321 21388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43305
I20260928 23:16:07.658357 21455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43305 every 8 connection(s)
I20260928 23:16:07.658674 21388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:07.659492 21456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:07.661688 21456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f: Bootstrap starting.
I20260928 23:16:07.662223 21456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:07.662468 21456 log.cc:824] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:07.663097 21456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f: No bootstrap required, opened a new log
I20260928 23:16:07.664601 21456 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260928 23:16:07.664733 21456 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:07.664764 21456 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe2ca5c36ec409d83fbac9f4948684f, State: Initialized, Role: FOLLOWER
I20260928 23:16:07.664886 21456 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260928 23:16:07.664945 21456 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:07.664973 21456 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:07.665019 21456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:07.665542 21456 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260928 23:16:07.665634 21456 leader_election.cc:302] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbe2ca5c36ec409d83fbac9f4948684f; no voters:
I20260928 23:16:07.665784 21456 leader_election.cc:288] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:07.665819 21459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:07.666002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21388
I20260928 23:16:07.666011 21459 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [term 1 LEADER]: Becoming Leader. State: Replica: cbe2ca5c36ec409d83fbac9f4948684f, State: Running, Role: LEADER
I20260928 23:16:07.666014 21456 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:07.666097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:07.666102 21459 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } }
I20260928 23:16:07.666290 392 external_mini_cluster.cc:1428] Setting key f5536ded817461338d5bbb06bc758e79
I20260928 23:16:07.666707 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } } }
I20260928 23:16:07.666787 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:07.667049 21461 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbe2ca5c36ec409d83fbac9f4948684f. Latest consensus state: current_term: 1 leader_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbe2ca5c36ec409d83fbac9f4948684f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43305 } } }
I20260928 23:16:07.667085 21469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:07.667124 21461 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:07.667570 21469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:07.669077 21469 catalog_manager.cc:1347] Generated new cluster ID: f5122ad315764d4ba7f27a03b341dc77
I20260928 23:16:07.669129 21469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:07.672361 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:07.666899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:59601 (local address 127.0.0.1:43305)
0928 23:16:07.667320 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:07.667331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:07.667460 (+ 129us) server_negotiation.cc:407] Connection header received
0928 23:16:07.667626 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:07.667633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:07.667698 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:07.667818 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:07.668013 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:07.669277 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:07.670356 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:07.670955 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:07.671043 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:07.671729 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:07.671733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:07.671733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:07.671927 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:07.671962 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:07.671975 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:07.672045 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:07.672135 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 23:16:07.672158 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":56,"threads_started":1}
I20260928 23:16:07.673331 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:07.678325 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:07.674822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:60017 (local address 127.0.0.1:43305)
0928 23:16:07.674866 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:07.674867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:07.675021 (+ 154us) server_negotiation.cc:407] Connection header received
0928 23:16:07.675095 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:07.675096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:07.675107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:07.675133 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:07.675311 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:07.676027 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:07.676999 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:07.677328 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:07.677362 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:07.678087 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:07.678088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:07.678088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:07.678092 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:07.678105 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:07.678109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:07.678136 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:07.678235 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 23:16:07.678248 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 23:16:07.705192 21469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:07.705974 21469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:07.710691 21469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbe2ca5c36ec409d83fbac9f4948684f: Generated new TSK 0
I20260928 23:16:07.710878 21469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:07.713742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21388
W20260928 23:16:07.717454 21373 connection.cc:535] client connection to 127.0.0.1:43305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43305): Connection reset by peer (error 104)
2026-09-28T23:16:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:07 dist-test-slave-6vgd krb5kdc[21487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:07 dist-test-slave-6vgd krb5kdc[21487](info): set up 2 sockets
Sep 28 23:16:07 dist-test-slave-6vgd krb5kdc[21487](info): commencing operation
krb5kdc: starting...
W20260928 23:16:08.661175 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:08 dist-test-slave-6vgd krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:08.786185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:08.865110 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:08.865357 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:08.865384 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:08.866489 21500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:08.866523 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:08.866536 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:08.866547 21500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:08.866560 21500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:08.867964 21500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:08.868148 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:08.868316 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:08.872761 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:08.872836 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:08.872905 21500 server_base.cc:1034] running on GCE node
W20260928 23:16:08.872948 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:08.873261 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:08.873554 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:08.874693 21500 hybrid_clock.cc:648] HybridClock initialized: now 1790637368874683 us; error 31 us; skew 500 ppm
Sep 28 23:16:08 dist-test-slave-6vgd krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:08.876940 21500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:08.877374 21500 webserver.cc:460] Webserver started at http://127.0.0.1:35315/ using document root <none> and password file <none>
I20260928 23:16:08.877575 21500 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:08.877621 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:08.877743 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:08.878754 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "c79f910d78454727a2faa99d1a10692b"
format_stamp: "Formatted at 2026-09-28 23:16:08 on dist-test-slave-6vgd"
server_key: "47bcfc1d367bd99d97dc612b7185b072"
server_key_iv: "3f4d4b843a637e6f40114bbff9c42fec"
server_key_version: "encryptionkey@0"
I20260928 23:16:08.879083 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "c79f910d78454727a2faa99d1a10692b"
format_stamp: "Formatted at 2026-09-28 23:16:08 on dist-test-slave-6vgd"
server_key: "47bcfc1d367bd99d97dc612b7185b072"
server_key_iv: "3f4d4b843a637e6f40114bbff9c42fec"
server_key_version: "encryptionkey@0"
I20260928 23:16:08.880339 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:08.880942 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:08.881105 21500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:08.881182 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "c79f910d78454727a2faa99d1a10692b"
format_stamp: "Formatted at 2026-09-28 23:16:08 on dist-test-slave-6vgd"
server_key: "47bcfc1d367bd99d97dc612b7185b072"
server_key_iv: "3f4d4b843a637e6f40114bbff9c42fec"
server_key_version: "encryptionkey@0"
I20260928 23:16:08.881274 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:08.884269 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:08.886200 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:08.886322 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:08.889822 21500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38491
I20260928 23:16:08.889850 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38491 every 8 connection(s)
I20260928 23:16:08.890129 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:08.891029 21568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:08.893180 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b: Bootstrap starting.
I20260928 23:16:08.893733 21568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:08.893975 21568 log.cc:824] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:08.894562 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b: No bootstrap required, opened a new log
I20260928 23:16:08.896037 21568 raft_consensus.cc:348] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 23:16:08.896168 21568 raft_consensus.cc:374] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:08.896200 21568 raft_consensus.cc:729] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79f910d78454727a2faa99d1a10692b, State: Initialized, Role: FOLLOWER
I20260928 23:16:08.896329 21568 consensus_queue.cc:260] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 23:16:08.896421 21568 raft_consensus.cc:388] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:08.896458 21568 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:08.896503 21568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:08.896981 21568 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 23:16:08.897066 21568 leader_election.cc:302] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c79f910d78454727a2faa99d1a10692b; no voters:
I20260928 23:16:08.897241 21568 leader_election.cc:288] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:08.897266 21571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:08.897392 21571 raft_consensus.cc:686] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [term 1 LEADER]: Becoming Leader. State: Replica: c79f910d78454727a2faa99d1a10692b, State: Running, Role: LEADER
I20260928 23:16:08.897492 21568 sys_catalog.cc:564] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:08.897476 21571 consensus_queue.cc:237] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } }
I20260928 23:16:08.897828 21573 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79f910d78454727a2faa99d1a10692b. Latest consensus state: current_term: 1 leader_uuid: "c79f910d78454727a2faa99d1a10692b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260928 23:16:08.897909 21573 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:08.897953 21572 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c79f910d78454727a2faa99d1a10692b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79f910d78454727a2faa99d1a10692b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38491 } } }
I20260928 23:16:08.898011 21572 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:08.898240 21578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:08.898733 21578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:08.900193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21500
I20260928 23:16:08.900313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:08.900497 392 external_mini_cluster.cc:1428] Setting key 6d96d6371c51f3b7bdf64b015baf9a58
I20260928 23:16:08.900681 21578 catalog_manager.cc:1347] Generated new cluster ID: ecb8a2086b054354b2f548fed6337d7e
I20260928 23:16:08.900729 21578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:08.902614 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:08.901027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:47527 (local address 127.0.0.1:38491)
0928 23:16:08.901295 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:08.901304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:08.901328 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:16:08.901430 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:08.901435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:08.901457 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:08.901567 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:08.902178 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:08.902181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:08.902181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:08.902185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:08.902221 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:08.902233 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:08.902285 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:08.902352 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 23:16:08.902487 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":77,"threads_started":1}
I20260928 23:16:08.903373 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:08.910265 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:08.905693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:43467 (local address 127.0.0.1:38491)
0928 23:16:08.905738 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:08.905740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:08.909402 (+ 3662us) server_negotiation.cc:407] Connection header received
0928 23:16:08.909471 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:08.909472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:08.909479 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:08.909522 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:08.910068 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:08.910069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:08.910069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:08.910073 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:08.910089 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:08.910093 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:08.910140 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:08.910184 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 23:16:08.910197 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 23:16:08.917353 21578 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:08.917526 21578 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:08.923383 21578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c79f910d78454727a2faa99d1a10692b: Generated new TSK 0
I20260928 23:16:08.923555 21578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:08.929114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21500
2026-09-28T23:16:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:08 dist-test-slave-6vgd krb5kdc[21599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:08 dist-test-slave-6vgd krb5kdc[21599](info): set up 2 sockets
Sep 28 23:16:08 dist-test-slave-6vgd krb5kdc[21599](info): commencing operation
krb5kdc: starting...
W20260928 23:16:09.868999 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:09 dist-test-slave-6vgd krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:09.977398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:10.055059 21612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:10.055243 21612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:10.055263 21612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:10.056370 21612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:10.056409 21612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:10.056424 21612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:10.056447 21612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:10.056478 21612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:10.057933 21612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:10.058144 21612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:10.058336 21612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:10.062731 21617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:10.062757 21618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:10.062747 21612 server_base.cc:1034] running on GCE node
W20260928 23:16:10.062731 21620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:10.063177 21612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:10.063422 21612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:10.064553 21612 hybrid_clock.cc:648] HybridClock initialized: now 1790637370064545 us; error 26 us; skew 500 ppm
Sep 28 23:16:10 dist-test-slave-6vgd krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637370, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:10.066787 21612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:10.067178 21612 webserver.cc:460] Webserver started at http://127.0.0.1:40861/ using document root <none> and password file <none>
I20260928 23:16:10.067369 21612 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:10.067411 21612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:10.067520 21612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:10.068470 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "d14f7021cf434c4280dad64b6d53db08"
format_stamp: "Formatted at 2026-09-28 23:16:10 on dist-test-slave-6vgd"
server_key: "c4853e135b896bd65d184be9b1503172"
server_key_iv: "ba8726d9f1f93c51fe6084844a215b3a"
server_key_version: "encryptionkey@0"
I20260928 23:16:10.068769 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "d14f7021cf434c4280dad64b6d53db08"
format_stamp: "Formatted at 2026-09-28 23:16:10 on dist-test-slave-6vgd"
server_key: "c4853e135b896bd65d184be9b1503172"
server_key_iv: "ba8726d9f1f93c51fe6084844a215b3a"
server_key_version: "encryptionkey@0"
I20260928 23:16:10.069949 21612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:16:10.070567 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:10.070715 21612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:10.070789 21612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "d14f7021cf434c4280dad64b6d53db08"
format_stamp: "Formatted at 2026-09-28 23:16:10 on dist-test-slave-6vgd"
server_key: "c4853e135b896bd65d184be9b1503172"
server_key_iv: "ba8726d9f1f93c51fe6084844a215b3a"
server_key_version: "encryptionkey@0"
I20260928 23:16:10.070863 21612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:10.091308 21612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:10.093338 21612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:10.093478 21612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:10.097146 21612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43189
I20260928 23:16:10.097186 21679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43189 every 8 connection(s)
I20260928 23:16:10.097568 21612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:10.098330 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:10.100463 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08: Bootstrap starting.
I20260928 23:16:10.100958 21680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:10.101197 21680 log.cc:824] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:10.101239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21612
I20260928 23:16:10.101333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:10.101502 392 external_mini_cluster.cc:1428] Setting key eeaf143971a341fc773261c39b7a1b58
I20260928 23:16:10.101888 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08: No bootstrap required, opened a new log
I20260928 23:16:10.103878 21680 raft_consensus.cc:348] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } }
I20260928 23:16:10.104040 21680 raft_consensus.cc:374] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:10.104075 21680 raft_consensus.cc:729] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14f7021cf434c4280dad64b6d53db08, State: Initialized, Role: FOLLOWER
I20260928 23:16:10.104192 21680 consensus_queue.cc:260] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } }
I20260928 23:16:10.104249 21680 raft_consensus.cc:388] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:10.104277 21680 raft_consensus.cc:482] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:10.104322 21680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:10.104813 21680 raft_consensus.cc:504] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } }
I20260928 23:16:10.104900 21680 leader_election.cc:302] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d14f7021cf434c4280dad64b6d53db08; no voters:
I20260928 23:16:10.105041 21680 leader_election.cc:288] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:10.105079 21685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:10.105237 21685 raft_consensus.cc:686] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [term 1 LEADER]: Becoming Leader. State: Replica: d14f7021cf434c4280dad64b6d53db08, State: Running, Role: LEADER
I20260928 23:16:10.105304 21680 sys_catalog.cc:564] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:10.105321 21685 consensus_queue.cc:237] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } }
I20260928 23:16:10.105758 21685 sys_catalog.cc:455] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d14f7021cf434c4280dad64b6d53db08. Latest consensus state: current_term: 1 leader_uuid: "d14f7021cf434c4280dad64b6d53db08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } } }
I20260928 23:16:10.105815 21686 sys_catalog.cc:455] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d14f7021cf434c4280dad64b6d53db08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d14f7021cf434c4280dad64b6d53db08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43189 } } }
I20260928 23:16:10.105872 21685 sys_catalog.cc:458] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:10.105954 21686 sys_catalog.cc:458] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:10.106184 21694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:10.106696 21694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:10.108381 21694 catalog_manager.cc:1347] Generated new cluster ID: 0988f0efe9c340d5aa11afcc45a1a32f
I20260928 23:16:10.108431 21694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:16:10 dist-test-slave-6vgd krb5kdc[21599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 23:16:10.112156 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:10.101939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:48943 (local address 127.0.0.1:43189)
0928 23:16:10.102085 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:10.102094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:10.102118 (+ 24us) server_negotiation.cc:407] Connection header received
0928 23:16:10.103014 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:10.103021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:10.103072 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:10.103180 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:10.103905 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:10.105243 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:10.106198 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:10.106769 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:10.106870 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:10.109263 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:10.109268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:10.109268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:10.109280 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:10.110744 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:10.111087 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:10.111090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:10.111091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:10.111169 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:10.111370 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:10.111371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:10.111371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:10.111593 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:10.111745 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:10.111825 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 23:16:10.112016 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260928 23:16:10.113194 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:10.115103 21694 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:10.115753 21694 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:10.119807 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:10.114380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:33653 (local address 127.0.0.1:43189)
0928 23:16:10.114414 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:10.114415 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:10.114529 (+ 114us) server_negotiation.cc:407] Connection header received
0928 23:16:10.114578 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:10.114578 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:10.114588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:10.114631 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:10.115206 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:10.115687 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:10.116344 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:10.116553 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:10.116605 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:10.117668 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:10.117669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:10.117670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:10.117676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:10.118832 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:10.119143 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:10.119144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:10.119144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:10.119193 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:10.119370 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:10.119370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:10.119371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:10.119468 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:10.119565 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:10.119673 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 23:16:10.119726 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":16}
I20260928 23:16:10.122488 21694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d14f7021cf434c4280dad64b6d53db08: Generated new TSK 0
I20260928 23:16:10.122642 21694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:10.126060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21612
W20260928 23:16:10.129767 21597 connection.cc:535] client connection to 127.0.0.1:43189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43189): Connection reset by peer (error 104)
2026-09-28T23:16:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:10 dist-test-slave-6vgd krb5kdc[21711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:10 dist-test-slave-6vgd krb5kdc[21711](info): set up 2 sockets
Sep 28 23:16:10 dist-test-slave-6vgd krb5kdc[21711](info): commencing operation
krb5kdc: starting...
W20260928 23:16:11.076924 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:11 dist-test-slave-6vgd krb5kdc[21711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:11.184324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:11.261466 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:11.261667 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:11.261688 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:11.262828 21724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:11.262871 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:11.262885 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:11.262908 21724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:11.262931 21724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:11.264343 21724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:11.264572 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:11.264772 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:11.269321 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:11.269345 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:11.269321 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:11.269522 21724 server_base.cc:1034] running on GCE node
I20260928 23:16:11.269713 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:11.269950 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:11.271095 21724 hybrid_clock.cc:648] HybridClock initialized: now 1790637371271082 us; error 26 us; skew 500 ppm
Sep 28 23:16:11 dist-test-slave-6vgd krb5kdc[21711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:11.276872 21724 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:11.277339 21724 webserver.cc:460] Webserver started at http://127.0.0.1:44163/ using document root <none> and password file <none>
I20260928 23:16:11.277549 21724 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:11.277596 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:11.277721 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:11.278793 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "bd3db6dcc2104e5db42f5a084e843e87"
format_stamp: "Formatted at 2026-09-28 23:16:11 on dist-test-slave-6vgd"
server_key: "4a19e099533b7f4f3f491b9c94be299a"
server_key_iv: "20a2fa86e63b9a65291b54b5565cd3a3"
server_key_version: "encryptionkey@0"
I20260928 23:16:11.279129 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "bd3db6dcc2104e5db42f5a084e843e87"
format_stamp: "Formatted at 2026-09-28 23:16:11 on dist-test-slave-6vgd"
server_key: "4a19e099533b7f4f3f491b9c94be299a"
server_key_iv: "20a2fa86e63b9a65291b54b5565cd3a3"
server_key_version: "encryptionkey@0"
I20260928 23:16:11.280298 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:11.280911 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:11.281059 21724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:11.281139 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "bd3db6dcc2104e5db42f5a084e843e87"
format_stamp: "Formatted at 2026-09-28 23:16:11 on dist-test-slave-6vgd"
server_key: "4a19e099533b7f4f3f491b9c94be299a"
server_key_iv: "20a2fa86e63b9a65291b54b5565cd3a3"
server_key_version: "encryptionkey@0"
I20260928 23:16:11.281227 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:11.295264 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:11.296977 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:11.297106 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:11.300825 21724 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36031
I20260928 23:16:11.300856 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36031 every 8 connection(s)
I20260928 23:16:11.301168 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:11.302101 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:11.304188 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87: Bootstrap starting.
I20260928 23:16:11.304677 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:11.304904 21792 log.cc:824] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:11.305517 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87: No bootstrap required, opened a new log
I20260928 23:16:11.306991 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 23:16:11.307122 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:11.307142 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3db6dcc2104e5db42f5a084e843e87, State: Initialized, Role: FOLLOWER
I20260928 23:16:11.307245 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 23:16:11.307291 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:11.307308 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:11.307337 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:11.307768 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 23:16:11.307852 21792 leader_election.cc:302] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd3db6dcc2104e5db42f5a084e843e87; no voters:
I20260928 23:16:11.307981 21792 leader_election.cc:288] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:11.308012 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:11.308123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21724
I20260928 23:16:11.308169 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [term 1 LEADER]: Becoming Leader. State: Replica: bd3db6dcc2104e5db42f5a084e843e87, State: Running, Role: LEADER
I20260928 23:16:11.308218 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:11.308236 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:11.308247 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } }
I20260928 23:16:11.308415 392 external_mini_cluster.cc:1428] Setting key 6033cab379115565156331b6be9403b0
I20260928 23:16:11.308539 21795 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd3db6dcc2104e5db42f5a084e843e87. Latest consensus state: current_term: 1 leader_uuid: "bd3db6dcc2104e5db42f5a084e843e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } } }
I20260928 23:16:11.308631 21795 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:11.308676 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd3db6dcc2104e5db42f5a084e843e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3db6dcc2104e5db42f5a084e843e87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36031 } } }
I20260928 23:16:11.308729 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:11.308943 21802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:11.309850 21802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:11.311342 21802 catalog_manager.cc:1347] Generated new cluster ID: 6d1a572b83e8490d9788b4378d22f8cb
I20260928 23:16:11.311395 21802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:11.314546 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:11.309186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:44389 (local address 127.0.0.1:36031)
0928 23:16:11.309348 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:11.309365 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:11.309404 (+ 39us) server_negotiation.cc:407] Connection header received
0928 23:16:11.309523 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:11.309530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:11.309601 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:11.309753 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:11.310000 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:11.311370 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:11.312496 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:11.313080 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:11.313175 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:11.313898 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:11.313902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:11.313902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:11.314099 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:11.314135 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:11.314148 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:11.314224 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:11.314317 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 23:16:11.314341 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":97,"threads_started":1}
I20260928 23:16:11.315424 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:11.317521 21802 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:11.318179 21802 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:11.320314 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:11.316800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:37319 (local address 127.0.0.1:36031)
0928 23:16:11.316835 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:11.316837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:11.316980 (+ 143us) server_negotiation.cc:407] Connection header received
0928 23:16:11.317024 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:11.317024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:11.317036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:11.317094 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:11.317400 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:11.318096 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:11.319080 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:11.319400 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:11.319464 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:11.320084 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:11.320085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:11.320086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:11.320090 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:11.320105 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:11.320110 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:11.320167 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:11.320220 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 23:16:11.320233 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:16:11.321283 21802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd3db6dcc2104e5db42f5a084e843e87: Generated new TSK 0
I20260928 23:16:11.321444 21802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:11.325562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21724
W20260928 23:16:11.329300 21709 connection.cc:535] client connection to 127.0.0.1:36031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36031): Connection reset by peer (error 104)
2026-09-28T23:16:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:11 dist-test-slave-6vgd krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:11 dist-test-slave-6vgd krb5kdc[21823](info): set up 2 sockets
Sep 28 23:16:11 dist-test-slave-6vgd krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260928 23:16:12.277024 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:12 dist-test-slave-6vgd krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 23:16:12.384178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:12.462247 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:12.462463 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:12.462495 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:12.463634 21836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:12.463680 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:12.463698 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:12.463713 21836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:12.463728 21836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:12.465231 21836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38813
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:12.465476 21836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:12.465677 21836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:12.470363 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:12.470373 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:12.470430 21836 server_base.cc:1034] running on GCE node
W20260928 23:16:12.470376 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:12.470847 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:12.471138 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:12.472306 21836 hybrid_clock.cc:648] HybridClock initialized: now 1790637372472288 us; error 32 us; skew 500 ppm
Sep 28 23:16:12 dist-test-slave-6vgd krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:12.474709 21836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 23:16:12.475116 21836 webserver.cc:460] Webserver started at http://127.0.0.1:38109/ using document root <none> and password file <none>
I20260928 23:16:12.475327 21836 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:12.475375 21836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:12.475497 21836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:12.476522 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "5652ea4f79b2453cb195a0ecf1031856"
format_stamp: "Formatted at 2026-09-28 23:16:12 on dist-test-slave-6vgd"
server_key: "d80c9df202f3c53475c3622ffb4cc2fc"
server_key_iv: "cf56ff192aa11513f58ce3d9ebdcf66b"
server_key_version: "encryptionkey@0"
I20260928 23:16:12.476853 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "5652ea4f79b2453cb195a0ecf1031856"
format_stamp: "Formatted at 2026-09-28 23:16:12 on dist-test-slave-6vgd"
server_key: "d80c9df202f3c53475c3622ffb4cc2fc"
server_key_iv: "cf56ff192aa11513f58ce3d9ebdcf66b"
server_key_version: "encryptionkey@0"
I20260928 23:16:12.478180 21836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:16:12.478921 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:12.479111 21836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:12.479197 21836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "5652ea4f79b2453cb195a0ecf1031856"
format_stamp: "Formatted at 2026-09-28 23:16:12 on dist-test-slave-6vgd"
server_key: "d80c9df202f3c53475c3622ffb4cc2fc"
server_key_iv: "cf56ff192aa11513f58ce3d9ebdcf66b"
server_key_version: "encryptionkey@0"
I20260928 23:16:12.479259 21836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:12.482577 21836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:12.484429 21836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:12.484552 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:12.488229 21836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38813
I20260928 23:16:12.488267 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38813 every 8 connection(s)
I20260928 23:16:12.488564 21836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:12.489498 21904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:12.491657 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856: Bootstrap starting.
I20260928 23:16:12.492189 21904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:12.492440 21904 log.cc:824] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:12.493036 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856: No bootstrap required, opened a new log
I20260928 23:16:12.494594 21904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260928 23:16:12.494727 21904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:12.494750 21904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5652ea4f79b2453cb195a0ecf1031856, State: Initialized, Role: FOLLOWER
I20260928 23:16:12.494879 21904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260928 23:16:12.494939 21904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:12.494961 21904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:12.494990 21904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:12.495474 21904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260928 23:16:12.495558 21904 leader_election.cc:302] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5652ea4f79b2453cb195a0ecf1031856; no voters:
I20260928 23:16:12.495704 21904 leader_election.cc:288] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:12.495748 21907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:12.495930 21904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:12.496074 21907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [term 1 LEADER]: Becoming Leader. State: Replica: 5652ea4f79b2453cb195a0ecf1031856, State: Running, Role: LEADER
I20260928 23:16:12.496254 21907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260928 23:16:12.496590 21908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5652ea4f79b2453cb195a0ecf1031856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260928 23:16:12.496600 21909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5652ea4f79b2453cb195a0ecf1031856. Latest consensus state: current_term: 1 leader_uuid: "5652ea4f79b2453cb195a0ecf1031856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5652ea4f79b2453cb195a0ecf1031856" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260928 23:16:12.496719 21908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:12.496738 21909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:12.496918 21916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:12.497418 21916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:12.498164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21836
I20260928 23:16:12.498289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:12.498445 392 external_mini_cluster.cc:1428] Setting key f226b7d828d9ef1e5fe94805d166e8d6
I20260928 23:16:12.498795 21916 catalog_manager.cc:1347] Generated new cluster ID: ae0c1a94ed6a4610b7161e963503d6e4
I20260928 23:16:12.498844 21916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:12.500509 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:12.499050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:40485 (local address 127.0.0.1:38813)
0928 23:16:12.499191 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:12.499200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:12.499233 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:16:12.499326 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:12.499331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:12.499353 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:12.499463 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:12.500115 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:12.500118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:12.500118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:12.500122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:12.500161 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:12.500172 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:12.500230 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:12.500294 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 23:16:12.500337 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260928 23:16:12.501537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:12.503897 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:12.502955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.92:33747 (local address 127.0.0.1:38813)
0928 23:16:12.502997 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:12.502998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:12.503101 (+ 103us) server_negotiation.cc:407] Connection header received
0928 23:16:12.503156 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:12.503157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:12.503162 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:12.503211 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:12.503699 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:12.503700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:12.503701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:12.503704 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:12.503716 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:12.503720 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:12.503777 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:12.503825 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 23:16:12.503838 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 23:16:12.515751 21916 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:12.515928 21916 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:12.532585 21916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5652ea4f79b2453cb195a0ecf1031856: Generated new TSK 0
I20260928 23:16:12.532842 21916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:12.542539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21836
2026-09-28T23:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:12 dist-test-slave-6vgd krb5kdc[21935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:12 dist-test-slave-6vgd krb5kdc[21935](info): set up 2 sockets
Sep 28 23:16:12 dist-test-slave-6vgd krb5kdc[21935](info): commencing operation
krb5kdc: starting...
W20260928 23:16:13.477012 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:13.583829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:13.662169 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:13.662384 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:13.662413 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:13.663563 21948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:13.663604 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:13.663621 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:13.663636 21948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:13.663651 21948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:13.665093 21948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:13.665354 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:13.665555 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:13.670266 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:13.670259 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:13.670255 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:13.670491 21948 server_base.cc:1034] running on GCE node
I20260928 23:16:13.670717 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:13.670971 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:13.672109 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790637373672088 us; error 34 us; skew 500 ppm
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:13.674463 21948 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:13.674861 21948 webserver.cc:460] Webserver started at http://127.0.98.62:37677/ using document root <none> and password file <none>
I20260928 23:16:13.675043 21948 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:13.675081 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:13.675166 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:13.676126 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "04f29726985c476885e5e83d3a204aa4"
format_stamp: "Formatted at 2026-09-28 23:16:13 on dist-test-slave-6vgd"
server_key: "f7608f272daadef8148a54289bfe4bc0"
server_key_iv: "e230fd219ab8cc707c04e02c95d37bd0"
server_key_version: "encryptionkey@0"
I20260928 23:16:13.676411 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "04f29726985c476885e5e83d3a204aa4"
format_stamp: "Formatted at 2026-09-28 23:16:13 on dist-test-slave-6vgd"
server_key: "f7608f272daadef8148a54289bfe4bc0"
server_key_iv: "e230fd219ab8cc707c04e02c95d37bd0"
server_key_version: "encryptionkey@0"
I20260928 23:16:13.677531 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:16:13.678117 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:13.678273 21948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:13.678345 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "04f29726985c476885e5e83d3a204aa4"
format_stamp: "Formatted at 2026-09-28 23:16:13 on dist-test-slave-6vgd"
server_key: "f7608f272daadef8148a54289bfe4bc0"
server_key_iv: "e230fd219ab8cc707c04e02c95d37bd0"
server_key_version: "encryptionkey@0"
I20260928 23:16:13.678406 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:13.688421 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:13.690354 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:13.690474 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:13.694002 21948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35331
I20260928 23:16:13.694037 22015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35331 every 8 connection(s)
I20260928 23:16:13.694324 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:13.695070 22016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:13.697122 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4: Bootstrap starting.
I20260928 23:16:13.697599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 21948
I20260928 23:16:13.697731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:13.697726 22016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:13.697904 392 external_mini_cluster.cc:1428] Setting key dd4aa50d0780f4d23ea07e02b1d461ea
I20260928 23:16:13.698017 22016 log.cc:824] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:13.698737 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4: No bootstrap required, opened a new log
I20260928 23:16:13.700891 22016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } }
I20260928 23:16:13.701057 22016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:13.701100 22016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f29726985c476885e5e83d3a204aa4, State: Initialized, Role: FOLLOWER
I20260928 23:16:13.701246 22016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } }
I20260928 23:16:13.701327 22016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:13.701361 22016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:13.701404 22016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:13.701997 22016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } }
I20260928 23:16:13.702090 22016 leader_election.cc:302] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04f29726985c476885e5e83d3a204aa4; no voters:
I20260928 23:16:13.702237 22016 leader_election.cc:288] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:13.702270 22021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:13.702420 22021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [term 1 LEADER]: Becoming Leader. State: Replica: 04f29726985c476885e5e83d3a204aa4, State: Running, Role: LEADER
I20260928 23:16:13.702471 22016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:13.702508 22021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } }
I20260928 23:16:13.702891 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f29726985c476885e5e83d3a204aa4. Latest consensus state: current_term: 1 leader_uuid: "04f29726985c476885e5e83d3a204aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } } }
I20260928 23:16:13.702992 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:13.702913 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f29726985c476885e5e83d3a204aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f29726985c476885e5e83d3a204aa4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35331 } } }
I20260928 23:16:13.703073 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:13.703229 22030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:13.703676 22030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:13.705147 22030 catalog_manager.cc:1347] Generated new cluster ID: cb04d8d877fd45cea96eb9dea8bfa6bf
I20260928 23:16:13.705248 22030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[21935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:13.709703 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:13.698349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58269 (local address 127.0.98.62:35331)
0928 23:16:13.698595 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:13.698602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:13.698624 (+ 22us) server_negotiation.cc:407] Connection header received
0928 23:16:13.699049 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:13.699056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:13.699102 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:13.699213 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:13.699922 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:13.700766 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:13.701803 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:13.702251 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:13.702650 (+ 399us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:13.706243 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:13.706248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:13.706248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:13.706259 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:13.708103 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:13.708497 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:13.708500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:13.708500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:13.708605 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:13.708810 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:13.708811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:13.708811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:13.709103 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:13.709284 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:13.709375 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 23:16:13.709482 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260928 23:16:13.710587 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:13.722667 22030 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:13.723249 22030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:13.729671 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:13.722622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36661 (local address 127.0.98.62:35331)
0928 23:16:13.722660 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:13.722662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:13.722909 (+ 247us) server_negotiation.cc:407] Connection header received
0928 23:16:13.722954 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:13.722955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:13.722964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:13.723025 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:13.723654 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:13.724376 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:13.725380 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:13.725676 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:13.725732 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:13.727058 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:13.727059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:13.727060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:13.727066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:13.728496 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:13.728913 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:13.728914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:13.728915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:13.728963 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:13.729148 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:13.729148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:13.729149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:13.729290 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:13.729382 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:13.729526 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 23:16:13.729583 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:16:13.730969 22030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f29726985c476885e5e83d3a204aa4: Generated new TSK 0
I20260928 23:16:13.731145 22030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:13.733184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 21948
W20260928 23:16:13.737282 21933 connection.cc:535] client connection to 127.0.98.62:35331 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35331): Connection reset by peer (error 104)
2026-09-28T23:16:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[22047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[22047](info): set up 2 sockets
Sep 28 23:16:13 dist-test-slave-6vgd krb5kdc[22047](info): commencing operation
krb5kdc: starting...
W20260928 23:16:14.677006 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:14 dist-test-slave-6vgd krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:14.788018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34757
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:14.866741 22060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:14.866942 22060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:14.866966 22060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:14.868090 22060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:14.868139 22060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:14.868163 22060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:14.868187 22060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:14.868209 22060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:14.869696 22060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:14.869918 22060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:14.870116 22060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:14.874732 22065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:14.874744 22068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:14.874713 22066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:14.874907 22060 server_base.cc:1034] running on GCE node
I20260928 23:16:14.875244 22060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:14.875527 22060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:14.876659 22060 hybrid_clock.cc:648] HybridClock initialized: now 1790637374876652 us; error 28 us; skew 500 ppm
Sep 28 23:16:14 dist-test-slave-6vgd krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637374, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:14.879009 22060 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:14.879433 22060 webserver.cc:460] Webserver started at http://127.0.98.62:43429/ using document root <none> and password file <none>
I20260928 23:16:14.879648 22060 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:14.879690 22060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:14.879818 22060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:14.880846 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "8577e9b64be04c67bda332ce45b57f39"
format_stamp: "Formatted at 2026-09-28 23:16:14 on dist-test-slave-6vgd"
server_key: "bb7a3a2bf8b470399c0591f9a69fc7be"
server_key_iv: "1c70b9429454c514382b580d226c585d"
server_key_version: "encryptionkey@0"
I20260928 23:16:14.881234 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "8577e9b64be04c67bda332ce45b57f39"
format_stamp: "Formatted at 2026-09-28 23:16:14 on dist-test-slave-6vgd"
server_key: "bb7a3a2bf8b470399c0591f9a69fc7be"
server_key_iv: "1c70b9429454c514382b580d226c585d"
server_key_version: "encryptionkey@0"
I20260928 23:16:14.882447 22060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:16:14.883098 22075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:14.883239 22060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:14.883311 22060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "8577e9b64be04c67bda332ce45b57f39"
format_stamp: "Formatted at 2026-09-28 23:16:14 on dist-test-slave-6vgd"
server_key: "bb7a3a2bf8b470399c0591f9a69fc7be"
server_key_iv: "1c70b9429454c514382b580d226c585d"
server_key_version: "encryptionkey@0"
I20260928 23:16:14.883363 22060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:14.906283 22060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:14.908010 22060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:14.908128 22060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:14.911691 22060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34757
I20260928 23:16:14.911731 22127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34757 every 8 connection(s)
I20260928 23:16:14.912043 22060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:14.912324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22060
I20260928 23:16:14.912446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:14.912628 392 external_mini_cluster.cc:1428] Setting key 91501001d29e5a13b62fbbd38cb5ed94
I20260928 23:16:14.912931 22128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:14.916123 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39: Bootstrap starting.
I20260928 23:16:14.916760 22128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:14.917057 22128 log.cc:824] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:14.917858 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39: No bootstrap required, opened a new log
I20260928 23:16:14.918608 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:14.913079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43389 (local address 127.0.98.62:34757)
0928 23:16:14.913295 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:14.913304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:14.913338 (+ 34us) server_negotiation.cc:407] Connection header received
0928 23:16:14.914006 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:14.914016 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:14.914091 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:14.914248 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:14.914388 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:14.915699 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:14.916558 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:14.917163 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:14.917316 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:14.918038 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:14.918041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:14.918042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:14.918193 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:14.918221 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:14.918232 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:14.918300 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:14.918402 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 23:16:14.918438 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260928 23:16:14.919590 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:14.919979 22128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } }
I20260928 23:16:14.920118 22128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:14.920151 22128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8577e9b64be04c67bda332ce45b57f39, State: Initialized, Role: FOLLOWER
I20260928 23:16:14.920267 22128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } }
I20260928 23:16:14.920337 22128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:14.920367 22128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:14.920413 22128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:14.920970 22128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } }
I20260928 23:16:14.921070 22128 leader_election.cc:302] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8577e9b64be04c67bda332ce45b57f39; no voters:
I20260928 23:16:14.921253 22128 leader_election.cc:288] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:14.921303 22138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:14.921450 22138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [term 1 LEADER]: Becoming Leader. State: Replica: 8577e9b64be04c67bda332ce45b57f39, State: Running, Role: LEADER
I20260928 23:16:14.921504 22128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:14.921618 22138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } }
I20260928 23:16:14.922000 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8577e9b64be04c67bda332ce45b57f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } } }
I20260928 23:16:14.922078 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:14.922242 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8577e9b64be04c67bda332ce45b57f39. Latest consensus state: current_term: 1 leader_uuid: "8577e9b64be04c67bda332ce45b57f39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8577e9b64be04c67bda332ce45b57f39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34757 } } }
I20260928 23:16:14.922298 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:14.922751 22145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:14.923233 22145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:14.924351 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:14.920882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39429 (local address 127.0.98.62:34757)
0928 23:16:14.920919 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:14.920921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:14.921036 (+ 115us) server_negotiation.cc:407] Connection header received
0928 23:16:14.921087 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:14.921087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:14.921098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:14.921145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:14.921356 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:14.922080 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:14.923163 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:14.923431 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:14.923490 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:14.924122 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:14.924124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:14.924124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:14.924129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:14.924144 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:14.924149 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:14.924209 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:14.924273 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 23:16:14.924282 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:16:14.924817 22145 catalog_manager.cc:1347] Generated new cluster ID: 006856afdf3d4e93a13541a6253da953
I20260928 23:16:14.924870 22145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:14.935911 22145 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:14.936457 22145 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:14.949348 22145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8577e9b64be04c67bda332ce45b57f39: Generated new TSK 0
I20260928 23:16:14.949556 22145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:14.952924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22060
W20260928 23:16:14.957141 22045 connection.cc:535] client connection to 127.0.98.62:34757 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34757): Connection reset by peer (error 104)
2026-09-28T23:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:15 dist-test-slave-6vgd krb5kdc[22159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:15 dist-test-slave-6vgd krb5kdc[22159](info): set up 2 sockets
Sep 28 23:16:15 dist-test-slave-6vgd krb5kdc[22159](info): commencing operation
krb5kdc: starting...
W20260928 23:16:15.897010 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:15 dist-test-slave-6vgd krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:16.005505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:16.083292 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:16.083487 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:16.083518 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:16.084633 22172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:16.084672 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:16.084690 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:16.084707 22172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:16.084721 22172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:16.086144 22172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:16.086370 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:16.086560 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:16.091022 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:16.091033 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:16.091039 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:16.091198 22172 server_base.cc:1034] running on GCE node
I20260928 23:16:16.091471 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:16.091735 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:16.092865 22172 hybrid_clock.cc:648] HybridClock initialized: now 1790637376092849 us; error 32 us; skew 500 ppm
Sep 28 23:16:16 dist-test-slave-6vgd krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637376, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:16.095176 22172 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:16.095563 22172 webserver.cc:460] Webserver started at http://127.0.98.62:44807/ using document root <none> and password file <none>
I20260928 23:16:16.095759 22172 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:16.095800 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:16.095912 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:16.096956 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "4bcd2330a63a4051a020247ea5436ec6"
format_stamp: "Formatted at 2026-09-28 23:16:16 on dist-test-slave-6vgd"
server_key: "c4bc054f6ccc84b25b29740feecae9be"
server_key_iv: "12992d2ff4cd575bb907f61e39eb5231"
server_key_version: "encryptionkey@0"
I20260928 23:16:16.097307 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bcd2330a63a4051a020247ea5436ec6"
format_stamp: "Formatted at 2026-09-28 23:16:16 on dist-test-slave-6vgd"
server_key: "c4bc054f6ccc84b25b29740feecae9be"
server_key_iv: "12992d2ff4cd575bb907f61e39eb5231"
server_key_version: "encryptionkey@0"
I20260928 23:16:16.098495 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 23:16:16.099207 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:16.099351 22172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:16.099424 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "4bcd2330a63a4051a020247ea5436ec6"
format_stamp: "Formatted at 2026-09-28 23:16:16 on dist-test-slave-6vgd"
server_key: "c4bc054f6ccc84b25b29740feecae9be"
server_key_iv: "12992d2ff4cd575bb907f61e39eb5231"
server_key_version: "encryptionkey@0"
I20260928 23:16:16.099484 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:16.102671 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:16.104403 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:16.104525 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:16.108182 22172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37959
I20260928 23:16:16.108218 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37959 every 8 connection(s)
I20260928 23:16:16.108558 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:16.109022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22172
I20260928 23:16:16.109134 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:16.109333 392 external_mini_cluster.cc:1428] Setting key ee962f6546e6ae9871035e25c4e0c394
I20260928 23:16:16.109575 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:16.111744 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:16.109771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34385 (local address 127.0.98.62:37959)
0928 23:16:16.109943 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:16.109952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:16.109975 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:16:16.110567 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:16.110574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:16.110602 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:16.110725 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:16.111377 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:16.111381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:16.111381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:16.111385 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:16.111424 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:16.111435 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:16.111489 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:16.111546 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 23:16:16.111586 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260928 23:16:16.112596 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:16.112613 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6: Bootstrap starting.
I20260928 23:16:16.113163 22240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:16.113485 22240 log.cc:824] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:16.114202 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6: No bootstrap required, opened a new log
I20260928 23:16:16.114848 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:16.113917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48105 (local address 127.0.98.62:37959)
0928 23:16:16.113954 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:16.113956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:16.114061 (+ 105us) server_negotiation.cc:407] Connection header received
0928 23:16:16.114116 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:16.114116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:16.114122 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:16.114173 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:16.114657 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:16.114659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:16.114659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:16.114663 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:16.114678 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:16.114682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:16.114744 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:16.114777 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:16:16.114788 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:16:16.116186 22240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } }
I20260928 23:16:16.116325 22240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:16.116348 22240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcd2330a63a4051a020247ea5436ec6, State: Initialized, Role: FOLLOWER
I20260928 23:16:16.116436 22240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } }
I20260928 23:16:16.116493 22240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:16.116518 22240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:16.116556 22240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:16.117057 22240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } }
I20260928 23:16:16.117139 22240 leader_election.cc:302] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bcd2330a63a4051a020247ea5436ec6; no voters:
I20260928 23:16:16.117316 22240 leader_election.cc:288] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:16.117391 22250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:16.117545 22250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcd2330a63a4051a020247ea5436ec6, State: Running, Role: LEADER
I20260928 23:16:16.117560 22240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:16.117631 22250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } }
I20260928 23:16:16.118000 22252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bcd2330a63a4051a020247ea5436ec6. Latest consensus state: current_term: 1 leader_uuid: "4bcd2330a63a4051a020247ea5436ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } } }
I20260928 23:16:16.117978 22251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bcd2330a63a4051a020247ea5436ec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bcd2330a63a4051a020247ea5436ec6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37959 } } }
I20260928 23:16:16.118153 22251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:16.118140 22252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:16.118503 22259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:16.118960 22259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:16.120329 22259 catalog_manager.cc:1347] Generated new cluster ID: 3e79d8aa0b80465abd0c7cdbe4f4eb62
I20260928 23:16:16.120379 22259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:16.132552 22259 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:16.132712 22259 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:16.134451 22259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bcd2330a63a4051a020247ea5436ec6: Generated new TSK 0
I20260928 23:16:16.134610 22259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:16.142057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22172
2026-09-28T23:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:16 dist-test-slave-6vgd krb5kdc[22271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:16 dist-test-slave-6vgd krb5kdc[22271](info): set up 2 sockets
Sep 28 23:16:16 dist-test-slave-6vgd krb5kdc[22271](info): commencing operation
krb5kdc: starting...
W20260928 23:16:17.081182 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:17.187347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:17.264720 22284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:17.264937 22284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:17.264967 22284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:17.266260 22284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:17.266320 22284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:17.266342 22284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:17.266359 22284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:17.266438 22284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:17.268055 22284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:17.268252 22284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:17.268420 22284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:17.273105 22292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:17.273123 22289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:17.273274 22284 server_base.cc:1034] running on GCE node
W20260928 23:16:17.273154 22290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:17.273576 22284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:17.273859 22284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:17.274992 22284 hybrid_clock.cc:648] HybridClock initialized: now 1790637377274984 us; error 28 us; skew 500 ppm
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:17.280213 22284 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:17.280633 22284 webserver.cc:460] Webserver started at http://127.0.98.62:45393/ using document root <none> and password file <none>
I20260928 23:16:17.280830 22284 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:17.280877 22284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:17.281005 22284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:17.282068 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "5c9b5e43c2e04013b56381a8e7ec0476"
format_stamp: "Formatted at 2026-09-28 23:16:17 on dist-test-slave-6vgd"
server_key: "757882bff49b42407fc71de3cf80ba93"
server_key_iv: "f33f12abadd88bc20bbd086f25d328c9"
server_key_version: "encryptionkey@0"
I20260928 23:16:17.282394 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c9b5e43c2e04013b56381a8e7ec0476"
format_stamp: "Formatted at 2026-09-28 23:16:17 on dist-test-slave-6vgd"
server_key: "757882bff49b42407fc71de3cf80ba93"
server_key_iv: "f33f12abadd88bc20bbd086f25d328c9"
server_key_version: "encryptionkey@0"
I20260928 23:16:17.283591 22284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:16:17.284240 22299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:17.284390 22284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:17.284467 22284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "5c9b5e43c2e04013b56381a8e7ec0476"
format_stamp: "Formatted at 2026-09-28 23:16:17 on dist-test-slave-6vgd"
server_key: "757882bff49b42407fc71de3cf80ba93"
server_key_iv: "f33f12abadd88bc20bbd086f25d328c9"
server_key_version: "encryptionkey@0"
I20260928 23:16:17.284533 22284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:17.292958 22284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:17.294649 22284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:17.294775 22284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:17.298523 22284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35419
I20260928 23:16:17.298555 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35419 every 8 connection(s)
I20260928 23:16:17.298868 22284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:17.299674 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:17.301261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22284
I20260928 23:16:17.301369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:17.301532 392 external_mini_cluster.cc:1428] Setting key 5f52a895deb1686a55ed37c9e5aa90b9
I20260928 23:16:17.302260 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476: Bootstrap starting.
I20260928 23:16:17.302951 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:17.303229 22352 log.cc:824] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:17.303974 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476: No bootstrap required, opened a new log
I20260928 23:16:17.306212 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 23:16:17.306382 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:17.306456 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9b5e43c2e04013b56381a8e7ec0476, State: Initialized, Role: FOLLOWER
I20260928 23:16:17.306591 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 23:16:17.306674 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:17.306715 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:17.306757 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:17.307322 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 23:16:17.307408 22352 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9b5e43c2e04013b56381a8e7ec0476; no voters:
I20260928 23:16:17.307559 22352 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:17.307605 22357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:17.307741 22357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9b5e43c2e04013b56381a8e7ec0476, State: Running, Role: LEADER
I20260928 23:16:17.307826 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:17.307898 22357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 23:16:17.308316 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 23:16:17.308395 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [sys.catalog]: This master's current role is: LEADER
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:17.308303 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9b5e43c2e04013b56381a8e7ec0476. Latest consensus state: current_term: 1 leader_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9b5e43c2e04013b56381a8e7ec0476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 23:16:17.308691 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:17.308911 22366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:17.309721 22366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:17.311288 22366 catalog_manager.cc:1347] Generated new cluster ID: 799c199706f246bf8c1063c2fbb7d1aa
I20260928 23:16:17.311335 22366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:17.312641 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:17.301990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48433 (local address 127.0.98.62:35419)
0928 23:16:17.302173 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:17.302183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:17.302216 (+ 33us) server_negotiation.cc:407] Connection header received
0928 23:16:17.302747 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:17.302756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:17.302818 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:17.302974 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:17.303593 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:17.304952 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:17.305746 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:17.306364 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:17.306475 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:17.308955 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:17.308959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:17.308960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:17.308973 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:17.310621 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:17.311003 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:17.311006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:17.311006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:17.311109 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:17.311341 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:17.311342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:17.311343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:17.311648 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:17.311773 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:17.312277 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 23:16:17.312430 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260928 23:16:17.313521 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:17.324186 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:17.317486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33923 (local address 127.0.98.62:35419)
0928 23:16:17.317524 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:17.317526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:17.317692 (+ 166us) server_negotiation.cc:407] Connection header received
0928 23:16:17.317725 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:17.317726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:17.317735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:17.317787 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:17.318361 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:17.319089 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:17.320015 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:17.320351 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:17.320417 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:17.321668 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:17.321670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:17.321670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:17.321677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:17.323146 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:17.323491 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:17.323491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:17.323492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:17.323548 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:17.323702 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:17.323703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:17.323703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:17.323826 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:17.323916 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:17.324013 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:16:17.324090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 23:16:17.331324 22366 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:17.331869 22366 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:17.341938 22366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9b5e43c2e04013b56381a8e7ec0476: Generated new TSK 0
I20260928 23:16:17.342160 22366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:17.349535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22284
2026-09-28T23:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22383](info): set up 2 sockets
Sep 28 23:16:17 dist-test-slave-6vgd krb5kdc[22383](info): commencing operation
krb5kdc: starting...
W20260928 23:16:18.297040 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:18 dist-test-slave-6vgd krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:18.406042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:18.482558 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:18.482748 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:18.482777 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:18.483882 22396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:18.483923 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:18.483942 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:18.483958 22396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:18.483973 22396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:18.485397 22396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:18.485615 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:18.485798 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:18.490212 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:18.490187 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:18.490219 22396 server_base.cc:1034] running on GCE node
W20260928 23:16:18.490195 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:18.490649 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:18.490917 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:18.492053 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790637378492038 us; error 27 us; skew 500 ppm
Sep 28 23:16:18 dist-test-slave-6vgd krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637378, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:18.494347 22396 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:18.494756 22396 webserver.cc:460] Webserver started at http://127.0.98.62:35559/ using document root <none> and password file <none>
I20260928 23:16:18.494938 22396 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:18.494983 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:18.495097 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:18.496061 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "69432d6a91714d799bca50b2009725ce"
format_stamp: "Formatted at 2026-09-28 23:16:18 on dist-test-slave-6vgd"
server_key: "d60cfd0bccbe9cdb78637f50fc19e96a"
server_key_iv: "0b5115a9110113225a3c55672086dce6"
server_key_version: "encryptionkey@0"
I20260928 23:16:18.496373 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "69432d6a91714d799bca50b2009725ce"
format_stamp: "Formatted at 2026-09-28 23:16:18 on dist-test-slave-6vgd"
server_key: "d60cfd0bccbe9cdb78637f50fc19e96a"
server_key_iv: "0b5115a9110113225a3c55672086dce6"
server_key_version: "encryptionkey@0"
I20260928 23:16:18.497560 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:18.498162 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:18.498303 22396 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:18.498378 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "69432d6a91714d799bca50b2009725ce"
format_stamp: "Formatted at 2026-09-28 23:16:18 on dist-test-slave-6vgd"
server_key: "d60cfd0bccbe9cdb78637f50fc19e96a"
server_key_iv: "0b5115a9110113225a3c55672086dce6"
server_key_version: "encryptionkey@0"
I20260928 23:16:18.498445 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:18.519515 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:18.521593 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:18.521725 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:18.525339 22396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42787
I20260928 23:16:18.525374 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42787 every 8 connection(s)
I20260928 23:16:18.525694 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:18.526473 22464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:18.528641 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce: Bootstrap starting.
I20260928 23:16:18.529134 22464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:18.529407 22464 log.cc:824] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:18.529928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22396
I20260928 23:16:18.529990 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce: No bootstrap required, opened a new log
I20260928 23:16:18.530035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:18.530196 392 external_mini_cluster.cc:1428] Setting key fc26d721e694b6f15249557ad633c340
I20260928 23:16:18.532694 22464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260928 23:16:18.532856 22464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:18.532886 22464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69432d6a91714d799bca50b2009725ce, State: Initialized, Role: FOLLOWER
I20260928 23:16:18.533002 22464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260928 23:16:18.533078 22464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:18.533118 22464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:18.533161 22464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:18.533686 22464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260928 23:16:18.533773 22464 leader_election.cc:302] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69432d6a91714d799bca50b2009725ce; no voters:
I20260928 23:16:18.533919 22464 leader_election.cc:288] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:18.533955 22469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:18.534083 22469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [term 1 LEADER]: Becoming Leader. State: Replica: 69432d6a91714d799bca50b2009725ce, State: Running, Role: LEADER
I20260928 23:16:18.534193 22464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:18.534216 22469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260928 23:16:18.534641 22470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69432d6a91714d799bca50b2009725ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } }
I20260928 23:16:18.534718 22470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:18.534960 22478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:18.535481 22478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:18.534968 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69432d6a91714d799bca50b2009725ce. Latest consensus state: current_term: 1 leader_uuid: "69432d6a91714d799bca50b2009725ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69432d6a91714d799bca50b2009725ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } }
I20260928 23:16:18.535852 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:18.536089 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:18.530627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33685 (local address 127.0.98.62:42787)
0928 23:16:18.530806 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:18.530816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:18.530848 (+ 32us) server_negotiation.cc:407] Connection header received
0928 23:16:18.531260 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:18.531269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:18.531331 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:18.531479 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:18.531620 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:18.532995 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:18.533842 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:18.534459 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:18.534575 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:18.535256 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:18.535261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:18.535261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:18.535467 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:18.535503 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:18.535518 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:18.535556 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:18.535819 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 23:16:18.535884 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 23:16:18.537093 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:18.537425 22478 catalog_manager.cc:1347] Generated new cluster ID: 199fe945d53d4988908a4de5750da0ca
I20260928 23:16:18.537475 22478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:18.541805 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:18.538550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45621 (local address 127.0.98.62:42787)
0928 23:16:18.538584 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:18.538586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:18.538718 (+ 132us) server_negotiation.cc:407] Connection header received
0928 23:16:18.538767 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:18.538768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:18.538787 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:18.538839 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:18.538960 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:18.539656 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:18.540624 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:18.540901 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:18.540955 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:18.541592 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:18.541594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:18.541594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:18.541598 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:18.541613 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:18.541617 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:18.541661 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:18.541719 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 23:16:18.541738 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:18.545004 22478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:18.545722 22478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:18.549154 22478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69432d6a91714d799bca50b2009725ce: Generated new TSK 0
I20260928 23:16:18.549326 22478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:18.551276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22396
2026-09-28T23:16:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:18 dist-test-slave-6vgd krb5kdc[22495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:18 dist-test-slave-6vgd krb5kdc[22495](info): set up 2 sockets
Sep 28 23:16:18 dist-test-slave-6vgd krb5kdc[22495](info): commencing operation
krb5kdc: starting...
W20260928 23:16:19.505024 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:19 dist-test-slave-6vgd krb5kdc[22495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:19.615260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40233
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:19.693898 22508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:19.694106 22508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:19.694137 22508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:19.695269 22508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:19.695312 22508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:19.695330 22508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:19.695345 22508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:19.695360 22508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:19.696827 22508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:19.697062 22508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:19.697279 22508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:19.701753 22516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:19.701756 22513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:19.701933 22508 server_base.cc:1034] running on GCE node
W20260928 23:16:19.701756 22514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:19.702193 22508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:19.702464 22508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:19.703601 22508 hybrid_clock.cc:648] HybridClock initialized: now 1790637379703588 us; error 31 us; skew 500 ppm
Sep 28 23:16:19 dist-test-slave-6vgd krb5kdc[22495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:19.705870 22508 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:19.706255 22508 webserver.cc:460] Webserver started at http://127.0.98.62:46625/ using document root <none> and password file <none>
I20260928 23:16:19.706465 22508 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:19.706511 22508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:19.706629 22508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:19.707664 22508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "df21eb3edcd4498c8b8e767c3c562957"
format_stamp: "Formatted at 2026-09-28 23:16:19 on dist-test-slave-6vgd"
server_key: "ab2eb6e8bfe5f654474d18abdca5e26c"
server_key_iv: "d2494fdb76d390138b9d6374bd5d7c92"
server_key_version: "encryptionkey@0"
I20260928 23:16:19.708007 22508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "df21eb3edcd4498c8b8e767c3c562957"
format_stamp: "Formatted at 2026-09-28 23:16:19 on dist-test-slave-6vgd"
server_key: "ab2eb6e8bfe5f654474d18abdca5e26c"
server_key_iv: "d2494fdb76d390138b9d6374bd5d7c92"
server_key_version: "encryptionkey@0"
I20260928 23:16:19.709200 22508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:19.709884 22523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:19.710029 22508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:19.710107 22508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "df21eb3edcd4498c8b8e767c3c562957"
format_stamp: "Formatted at 2026-09-28 23:16:19 on dist-test-slave-6vgd"
server_key: "ab2eb6e8bfe5f654474d18abdca5e26c"
server_key_iv: "d2494fdb76d390138b9d6374bd5d7c92"
server_key_version: "encryptionkey@0"
I20260928 23:16:19.710173 22508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:19.713240 22508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:19.714881 22508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:19.715004 22508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:19.718638 22508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40233
I20260928 23:16:19.718668 22575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40233 every 8 connection(s)
I20260928 23:16:19.718946 22508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:19.719259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22508
I20260928 23:16:19.719377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:19.719523 392 external_mini_cluster.cc:1428] Setting key 81049cc295cfdc7e6d673281f68fc846
I20260928 23:16:19.719990 22576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:19.721760 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:19.719968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51137 (local address 127.0.98.62:40233)
0928 23:16:19.720154 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:19.720163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:19.720190 (+ 27us) server_negotiation.cc:407] Connection header received
0928 23:16:19.720609 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:19.720615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:19.720638 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:19.720756 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:19.721414 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:19.721417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:19.721418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:19.721422 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:19.721458 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:19.721467 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:19.721519 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:19.721575 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 23:16:19.721613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260928 23:16:19.722607 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:19.722903 22576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: Bootstrap starting.
I20260928 23:16:19.723430 22576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:19.723697 22576 log.cc:824] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:19.724390 22576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: No bootstrap required, opened a new log
I20260928 23:16:19.724759 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:19.723826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59035 (local address 127.0.98.62:40233)
0928 23:16:19.723867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:19.723869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:19.723948 (+ 79us) server_negotiation.cc:407] Connection header received
0928 23:16:19.724015 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:19.724015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:19.724018 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:19.724063 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:19.724578 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:19.724580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:19.724580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 23:16:19.724584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:19.724597 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 23:16:19.724601 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:19.724660 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:19.724693 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 23:16:19.724704 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:16:19.726318 22576 raft_consensus.cc:348] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } }
I20260928 23:16:19.726449 22576 raft_consensus.cc:374] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 23:16:19.726470 22576 raft_consensus.cc:729] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df21eb3edcd4498c8b8e767c3c562957, State: Initialized, Role: FOLLOWER
I20260928 23:16:19.726581 22576 consensus_queue.cc:260] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } }
I20260928 23:16:19.726655 22576 raft_consensus.cc:388] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 23:16:19.726686 22576 raft_consensus.cc:482] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 23:16:19.726743 22576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:19.727232 22576 raft_consensus.cc:504] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } }
I20260928 23:16:19.727325 22576 leader_election.cc:302] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df21eb3edcd4498c8b8e767c3c562957; no voters:
I20260928 23:16:19.727485 22576 leader_election.cc:288] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 23:16:19.727516 22586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:19.727643 22586 raft_consensus.cc:686] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [term 1 LEADER]: Becoming Leader. State: Replica: df21eb3edcd4498c8b8e767c3c562957, State: Running, Role: LEADER
I20260928 23:16:19.727710 22576 sys_catalog.cc:564] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:19.727752 22586 consensus_queue.cc:237] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } }
I20260928 23:16:19.728122 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df21eb3edcd4498c8b8e767c3c562957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } } }
I20260928 23:16:19.728137 22588 sys_catalog.cc:455] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df21eb3edcd4498c8b8e767c3c562957. Latest consensus state: current_term: 1 leader_uuid: "df21eb3edcd4498c8b8e767c3c562957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df21eb3edcd4498c8b8e767c3c562957" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40233 } } }
I20260928 23:16:19.728299 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:19.728322 22588 sys_catalog.cc:458] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957 [sys.catalog]: This master's current role is: LEADER
W20260928 23:16:19.729240 22602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:19.729291 22602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:19.729338 22595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:19.729475 22595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:19.730868 22595 catalog_manager.cc:1347] Generated new cluster ID: 3be23200a49c40ed8b29fb1790789989
I20260928 23:16:19.730914 22595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:19.737764 22595 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:19.737907 22595 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:19.750191 22595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df21eb3edcd4498c8b8e767c3c562957: Generated new TSK 0
I20260928 23:16:19.750375 22595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 23:16:19.755245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22508
2026-09-28T23:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1203 ms)
[----------] 18 tests from AuthTokenIssuingTest (21617 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:19 dist-test-slave-6vgd krb5kdc[22607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:19 dist-test-slave-6vgd krb5kdc[22607](info): set up 2 sockets
Sep 28 23:16:19 dist-test-slave-6vgd krb5kdc[22607](info): commencing operation
krb5kdc: starting...
W20260928 23:16:20.701011 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:20 dist-test-slave-6vgd krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:20.812428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43629
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:20.890033 22620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:20.890242 22620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:20.890259 22620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:20.890277 22620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:20.891390 22620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:20.891438 22620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:20.891451 22620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:20.891462 22620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:20.891472 22620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:20.892954 22620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:20.893183 22620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:20.893424 22620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:20.897841 22625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:20.897956 22620 server_base.cc:1034] running on GCE node
W20260928 23:16:20.897976 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:20.897877 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:20.898253 22620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:20.898521 22620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:20.899652 22620 hybrid_clock.cc:648] HybridClock initialized: now 1790637380899644 us; error 26 us; skew 500 ppm
Sep 28 23:16:20 dist-test-slave-6vgd krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:20.901950 22620 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:20.902338 22620 webserver.cc:460] Webserver started at http://127.0.98.62:32913/ using document root <none> and password file <none>
I20260928 23:16:20.902521 22620 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:20.902559 22620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:20.902649 22620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:20.903620 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "f959683b912d483687eb87722d9b323b"
format_stamp: "Formatted at 2026-09-28 23:16:20 on dist-test-slave-6vgd"
server_key: "4cc129adaf88ba1d46b9a0580d62eda4"
server_key_iv: "04e8a2fc83c067561ceacc11702c646a"
server_key_version: "encryptionkey@0"
I20260928 23:16:20.903900 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "f959683b912d483687eb87722d9b323b"
format_stamp: "Formatted at 2026-09-28 23:16:20 on dist-test-slave-6vgd"
server_key: "4cc129adaf88ba1d46b9a0580d62eda4"
server_key_iv: "04e8a2fc83c067561ceacc11702c646a"
server_key_version: "encryptionkey@0"
I20260928 23:16:20.905014 22620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 23:16:20.905674 22635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:20.905812 22620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:20.905900 22620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "f959683b912d483687eb87722d9b323b"
format_stamp: "Formatted at 2026-09-28 23:16:20 on dist-test-slave-6vgd"
server_key: "4cc129adaf88ba1d46b9a0580d62eda4"
server_key_iv: "04e8a2fc83c067561ceacc11702c646a"
server_key_version: "encryptionkey@0"
I20260928 23:16:20.905954 22620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:20.918036 22620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:20.919605 22620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:20.919708 22620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:20.923280 22620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43629
I20260928 23:16:20.923311 22687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43629 every 8 connection(s)
I20260928 23:16:20.923604 22620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:20.924479 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:20.925980 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:20.926594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22620
I20260928 23:16:20.926682 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:20.926841 392 external_mini_cluster.cc:1428] Setting key 66eb038785a290376c938a722748c78e
Sep 28 23:16:20 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 23:16:20.934602 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.926577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43629 (local address 127.0.0.1:52698)
0928 23:16:20.926718 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:20.926759 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:20.927250 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:20.927519 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:20.927521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:20.927754 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:20.927915 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:20.927919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:20.928874 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:20.928875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:20.929566 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:20.929568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:20.930023 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:20.930024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:20.930084 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:20.930554 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:20.930562 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:20.931796 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:20.933336 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:20.933342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:20.933344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:20.933514 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:20.933706 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:20.933706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:20.933706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:20.933780 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:20.934079 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:20.934083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:20.934141 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 23:16:20.934218 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:16:20.934427 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":78,"thread_start_us":67,"threads_started":1}
I20260928 23:16:20.934604 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.926748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.0.98.62:43629)
0928 23:16:20.926892 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:20.926897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:20.926937 (+ 40us) server_negotiation.cc:407] Connection header received
0928 23:16:20.927398 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:20.927401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:20.927422 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:20.927519 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:20.928007 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:20.928795 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:20.929636 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:20.929941 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:20.930031 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:20.931866 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:20.931872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:20.931873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:20.931879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:20.933278 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:20.933579 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:20.933581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:20.933582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:20.933654 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:20.933829 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:20.933830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:20.933830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:20.934023 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:20.934093 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:20.934248 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 23:16:20.934372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 23:16:20.935572 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:20.935877 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.935768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37406)
0928 23:16:20.935808 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:20.935854 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 23:16:20.936008 22636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
I20260928 23:16:20.936344 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.936267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37418)
0928 23:16:20.936301 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:20.936323 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 23:16:20.936430 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:16:20.948144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38873
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:16:20.985621 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } attempt: 1
I20260928 23:16:20.986114 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.985982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37422)
0928 23:16:20.986042 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:20.986085 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:16:20.986456 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:20.986378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37430)
0928 23:16:20.986414 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:20.986439 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:16:20.986548 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
W20260928 23:16:21.026423 22695 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:21.026631 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:21.026654 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:21.026681 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:21.027792 22695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:21.027834 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:21.027854 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:21.027870 22695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:21.027886 22695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:21.029352 22695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:21.029575 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:21.029767 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:21.034291 22701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:21.034423 22703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:21.034370 22700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:21.034293 22695 server_base.cc:1034] running on GCE node
I20260928 23:16:21.034706 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:21.034977 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:21.036128 22695 hybrid_clock.cc:648] HybridClock initialized: now 1790637381036116 us; error 27 us; skew 500 ppm
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:21.038506 22695 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:16:21.038899 22695 webserver.cc:460] Webserver started at http://127.0.98.61:39349/ using document root <none> and password file <none>
I20260928 23:16:21.039086 22695 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:21.039134 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:21.039254 22695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:21.040258 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data/instance:
uuid: "898874d060164d16abc206a8f32c0bac"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "360d47e931c4c606c729b429a4fdcc84"
server_key_iv: "4e2f8f5575cc86650cdd80b1cbac05b1"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.040578 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal/instance:
uuid: "898874d060164d16abc206a8f32c0bac"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "360d47e931c4c606c729b429a4fdcc84"
server_key_iv: "4e2f8f5575cc86650cdd80b1cbac05b1"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.041962 22695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 23:16:21.042609 22710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:21.042760 22695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 23:16:21.042837 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "898874d060164d16abc206a8f32c0bac"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "360d47e931c4c606c729b429a4fdcc84"
server_key_iv: "4e2f8f5575cc86650cdd80b1cbac05b1"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.042898 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:21.054692 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } attempt: 2
I20260928 23:16:21.054905 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:21.055073 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.054965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37438)
0928 23:16:21.055012 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.055043 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 23:16:21.055584 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.055497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37446)
0928 23:16:21.055532 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.055557 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:16:21.055658 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38873: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38873: connect: Connection refused (error 111)
I20260928 23:16:21.056777 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:21.056900 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:21.060765 22695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38873
I20260928 23:16:21.060868 22762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38873 every 8 connection(s)
I20260928 23:16:21.061120 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:16:21.061986 22763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:21.062263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22695
I20260928 23:16:21.062347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-1/wal/instance
I20260928 23:16:21.062497 392 external_mini_cluster.cc:1428] Setting key 1c276dc31beeec2ced039e038ed7e6ae
I20260928 23:16:21.063542 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 23:16:21.073045 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.064205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.98.62:43629)
0928 23:16:21.064272 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.064274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.064506 (+ 232us) server_negotiation.cc:407] Connection header received
0928 23:16:21.065099 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.065101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.065114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.065154 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.065980 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.066462 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.067882 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.068076 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.068120 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.070383 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.070385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.070385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.070391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.071654 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.072108 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.072109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.072109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.072171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.072441 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.072442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.072442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.072548 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.072606 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.072868 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.072953 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 23:16:21.073191 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.064176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43629 (local address 127.0.0.1:52704)
0928 23:16:21.064367 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.064415 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.064991 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.065278 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.065282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.065605 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.065895 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.065902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.066551 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.066553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.067795 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.067798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.068159 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.068160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.068320 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.068818 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.068827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.070298 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.071794 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.071817 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.071819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.072041 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.072248 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.072249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.072249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.072379 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.072610 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.072617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.072748 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.072840 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.072985 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260928 23:16:21.073848 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:21.080682 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.074097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37460)
0928 23:16:21.074143 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.074152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.074189 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.074395 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.074395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.074520 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.074560 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.074561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.075719 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.075720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.076201 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.076202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.076678 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.076679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.076745 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.077117 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.077120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.078447 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.079918 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.079919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.079919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.080060 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.080207 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.080207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.080207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.080246 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.080469 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.080469 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.080502 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.080548 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.080620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:16:21.080776 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.074134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.0.98.61:38873)
0928 23:16:21.074286 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.074290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.074304 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:16:21.074319 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.074321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.074330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.074389 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.074619 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.075626 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.076272 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.076609 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.076673 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.078522 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.078529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.078529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.078535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.079850 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.080118 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.080120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.080120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.080165 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.080289 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.080290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.080290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.080428 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.080486 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.080569 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.080673 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260928 23:16:21.081254 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:21.081568 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.081451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55102)
0928 23:16:21.081495 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.081544 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:16:21.081696 22712 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
I20260928 23:16:21.082056 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.081990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55104)
0928 23:16:21.082020 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.082041 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 23:16:21.082139 22763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40591: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:16:21.085589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40591
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:16:21.131371 22763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } attempt: 1
I20260928 23:16:21.131881 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.131742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55118)
0928 23:16:21.131798 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.131845 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:16:21.132200 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.132115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55124)
0928 23:16:21.132151 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.132174 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 23:16:21.132300 22763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40591: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
I20260928 23:16:21.146799 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } attempt: 3
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:21.153867 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.147180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37474)
0928 23:16:21.147302 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.147317 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.147376 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.147520 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.147521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.147669 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.147741 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.147742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.148604 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.148605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.149125 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.149126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.149602 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.149602 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.149660 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.150085 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.150089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.151435 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.153009 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.153010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.153010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.153162 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.153367 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.153368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.153368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.153416 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.153643 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.153644 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.153681 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.153731 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.153801 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 23:16:21.153926 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.147286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.0.98.61:38873)
0928 23:16:21.147351 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.147354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.147366 (+ 12us) server_negotiation.cc:407] Connection header received
0928 23:16:21.147434 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.147435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.147450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.147505 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.147799 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.148528 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.149199 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.149528 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.149597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.151521 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.151523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.151523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.151530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.152936 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.153246 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.153247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.153247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.153308 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.153462 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.153463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.153463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.153591 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.153656 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.153757 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.153838 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 23:16:21.154596 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:21.154919 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.154832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55130)
0928 23:16:21.154870 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.154898 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 23:16:21.155229 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.155168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55136)
0928 23:16:21.155191 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.155215 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 23:16:21.155309 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40591: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
W20260928 23:16:21.166118 22770 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:21.166343 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:21.166358 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:21.166378 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:21.167462 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:21.167496 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:21.167508 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:21.167520 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:21.167534 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:21.168959 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43629,127.0.98.61:38873,127.0.98.60:40591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:21.169149 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:21.169397 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:21.174039 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:21.174027 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:21.174075 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:21.174193 22770 server_base.cc:1034] running on GCE node
I20260928 23:16:21.174499 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:21.174773 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:21.175933 22770 hybrid_clock.cc:648] HybridClock initialized: now 1790637381175912 us; error 35 us; skew 500 ppm
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:21.178508 22770 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:16:21.178941 22770 webserver.cc:460] Webserver started at http://127.0.98.60:37195/ using document root <none> and password file <none>
I20260928 23:16:21.179178 22770 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:21.179229 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:21.179349 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:21.180420 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data/instance:
uuid: "e1751526ff57494986a337cee094e77c"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "ef5f3c38e64620f5afe8d641181d1998"
server_key_iv: "19ba7966a29f0e6495fa7af876cd5fb6"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.180737 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal/instance:
uuid: "e1751526ff57494986a337cee094e77c"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "ef5f3c38e64620f5afe8d641181d1998"
server_key_iv: "19ba7966a29f0e6495fa7af876cd5fb6"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.182008 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:21.182693 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:21.182834 22770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:21.182911 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "e1751526ff57494986a337cee094e77c"
format_stamp: "Formatted at 2026-09-28 23:16:21 on dist-test-slave-6vgd"
server_key: "ef5f3c38e64620f5afe8d641181d1998"
server_key_iv: "19ba7966a29f0e6495fa7af876cd5fb6"
server_key_version: "encryptionkey@0"
I20260928 23:16:21.182971 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:21.186448 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } attempt: 1
I20260928 23:16:21.186796 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.186684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55148)
0928 23:16:21.186733 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.186764 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:16:21.187083 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.187007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55160)
0928 23:16:21.187040 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.187065 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 23:16:21.187153 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40591: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40591: connect: Connection refused (error 111)
I20260928 23:16:21.194445 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:21.196130 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:21.196257 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:21.199922 22770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40591
I20260928 23:16:21.199951 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40591 every 8 connection(s)
I20260928 23:16:21.200269 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:16:21.200438 22763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } attempt: 2
I20260928 23:16:21.201319 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:21.202893 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:21.208796 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.200716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55170)
0928 23:16:21.200774 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.200786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.200829 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.201688 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.201689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.201853 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.201903 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.201905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.202953 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.202954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.203648 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.203652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.204276 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.204277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.204322 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.204687 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.204691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.206204 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.207700 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.207701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.207702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.207872 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.208067 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.208068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.208068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.208129 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.208486 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.208487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.208537 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.208603 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.208698 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 23:16:21.209105 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.200843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.0.98.60:40591)
0928 23:16:21.200998 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.201006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.201029 (+ 23us) server_negotiation.cc:407] Connection header received
0928 23:16:21.201507 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.201515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.201563 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.201677 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.201985 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.202860 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.203730 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.204184 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.204311 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.206280 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.206286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.206287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.206295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.207616 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.207944 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.207946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.207947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.208017 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.208180 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.208181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.208182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.208423 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.208485 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.208671 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.208796 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260928 23:16:21.209844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 22770
I20260928 23:16:21.209928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790637099107577-392-0/minicluster-data/master-2/wal/instance
I20260928 23:16:21.210093 392 external_mini_cluster.cc:1428] Setting key c5751612cc6c0adf85c2fc6b323733b2
I20260928 23:16:21.210788 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: Bootstrap starting.
I20260928 23:16:21.210959 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.203553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.0.98.62:43629)
0928 23:16:21.203600 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.203602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.203720 (+ 118us) server_negotiation.cc:407] Connection header received
0928 23:16:21.203777 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.203778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.203787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.203851 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.204218 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.204837 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.205741 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.205918 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.205968 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.207881 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.207882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.207882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.207889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.209228 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.209602 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.209603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.209603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.209664 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.210387 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.210388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.210389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.210519 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.210561 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.210792 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.210868 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 23:16:21.211237 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.203479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43629 (local address 127.0.0.1:52720)
0928 23:16:21.203625 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.203667 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.203727 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.203865 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.203867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.204091 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.204154 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.204155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.204931 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.204932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.205627 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.205629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.205992 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.205993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.206039 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.206455 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.206460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.207813 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.209301 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.209306 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.209308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.209521 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.210245 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.210246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.210247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.210308 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.210605 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.210612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.210666 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.211024 (+ 358us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.211135 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260928 23:16:21.211735 22763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:21.211998 22763 log.cc:824] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:21.212873 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: No bootstrap required, opened a new log
I20260928 23:16:21.213424 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:21.214891 22763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.215031 22763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898874d060164d16abc206a8f32c0bac, State: Initialized, Role: FOLLOWER
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:16:21.215139 22763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.215427 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.215492 22763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:21.215590 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:21.218685 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.210410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.60:40591)
0928 23:16:21.210443 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.210445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.210659 (+ 214us) server_negotiation.cc:407] Connection header received
0928 23:16:21.210707 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.210707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.210716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.210764 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.211455 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.211930 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.212907 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.213123 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.213176 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.215690 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.215692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.215692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.215699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.217324 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.217672 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.217673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.217673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.217724 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.217907 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.217908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.217909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.218022 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.218156 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.218533 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.218599 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:16:21.219256 22855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:21.219352 22855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:21.220000 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.213742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38873 (local address 127.0.0.1:37478)
0928 23:16:21.213780 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.213789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.213820 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.213919 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.213920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.214042 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.214080 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.214082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.214647 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.214648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.215152 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.215153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.215621 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.215622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.215702 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.216030 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.216034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.217399 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.219021 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.219022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.219022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.219211 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.219403 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.219404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.219404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.219459 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.219722 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.219723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.219764 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.219832 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.219916 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 23:16:21.217895 22695 master_runner.cc:186] Error getting master registration for 127.0.98.62:43629: OK, instance_id { permanent_uuid: "f959683b912d483687eb87722d9b323b" instance_seqno: 1790637380922305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:16:21.220021 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.213802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.61:38873)
0928 23:16:21.213837 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.213839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.213847 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:16:21.213865 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.213866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.213874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.213917 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.214146 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.214578 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.215238 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.215512 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.215552 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.217477 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.217478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.217478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.217484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.218951 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.219284 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.219285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.219285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.219338 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.219514 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.219515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.219515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.219664 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.219726 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.219876 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.219946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 23:16:21.220070 22695 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:16:21.220311 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637381, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:21.226490 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.220538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55184)
0928 23:16:21.220628 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.220636 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.220670 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.220813 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.220814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.220929 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.220951 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.220952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.221555 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.221555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.222297 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.222300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.222666 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.222667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.222714 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.223054 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.223059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.224141 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.225540 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.225541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.225541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.225767 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.225954 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.225954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.225955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.226001 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.226245 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.226246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.226293 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.226355 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.226429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 23:16:21.226593 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.220602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55184 (local address 127.0.98.60:40591)
0928 23:16:21.220723 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.220725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.220730 (+ 5us) server_negotiation.cc:407] Connection header received
0928 23:16:21.220744 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.220744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.220751 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.220785 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.221013 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.221494 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.222375 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.222610 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.222658 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.224209 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.224211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.224211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.224217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.225473 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.225846 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.225847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.225847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.225897 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.226056 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.226056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.226057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.226183 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.226245 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.226396 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.226490 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":108}
I20260928 23:16:21.228250 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: Bootstrap starting.
I20260928 23:16:21.228596 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.220148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.0.98.62:43629)
0928 23:16:21.220193 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.220195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.220202 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:16:21.220222 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.220222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.220232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.220286 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.220848 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.221609 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.222506 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.222788 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.222850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.225472 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.225474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.225474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.225481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.227158 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.227528 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.227529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.227530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.227587 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.227787 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.227788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.227788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.227911 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.228055 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.228434 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.228510 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:16:21.229131 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:21.229444 22838 log.cc:824] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:21.230233 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: No bootstrap required, opened a new log
I20260928 23:16:21.232189 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.232343 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1751526ff57494986a337cee094e77c, State: Initialized, Role: FOLLOWER
I20260928 23:16:21.232471 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.232769 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.232842 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:21.232916 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260928 23:16:21.234773 22869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:21.234817 22869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:21.237931 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.229570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.98.61:38873)
0928 23:16:21.229652 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.229654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.229662 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:16:21.229683 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.229684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.229693 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.229757 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.230299 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.230995 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.232053 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.232343 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.232406 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.235026 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.235027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.235027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.235033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.236462 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.236832 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.236834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.236834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.236892 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.237073 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.237073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.237074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.237190 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.237388 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.237773 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.237846 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:16:21.235584 22770 master_runner.cc:186] Error getting master registration for 127.0.98.62:43629: OK, instance_id { permanent_uuid: "f959683b912d483687eb87722d9b323b" instance_seqno: 1790637380922305 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:16:21.238528 22770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:16:21.239324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:21.240139 22662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f959683b912d483687eb87722d9b323b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52734
I20260928 23:16:21.262264 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } attempt: 2
Sep 28 23:16:21 dist-test-slave-6vgd krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:16:21.269832 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.262623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40591 (local address 127.0.0.1:55198)
0928 23:16:21.262671 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:21.262685 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:21.262734 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:21.262963 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:21.262964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:21.263132 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:21.263176 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.263178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.264099 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.264100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.264909 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:21.264910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.265264 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.265265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:21.265317 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.265765 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:21.265770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:21.267221 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:21.268898 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.268899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.268899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.269058 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.269237 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:21.269237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:21.269238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:21.269293 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:21.269519 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:21.269519 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:21.269578 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 23:16:21.269636 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 23:16:21.269733 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 23:16:21.269832 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.262794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.98.60:40591)
0928 23:16:21.262852 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.262853 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.262863 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:16:21.262890 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.262891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.262898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.262946 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.263271 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.264005 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.264983 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.265178 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.265266 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.267303 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.267305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.267305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.267313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.268830 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.269116 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.269117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.269118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.269172 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.269342 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.269344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.269344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.269473 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.269530 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.269666 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.269738 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:16:21.271423 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: Bootstrap starting.
I20260928 23:16:21.272254 22688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:21.272554 22688 log.cc:824] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:21.273352 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: No bootstrap required, opened a new log
I20260928 23:16:21.274667 22688 raft_consensus.cc:348] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.274816 22688 raft_consensus.cc:729] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f959683b912d483687eb87722d9b323b, State: Initialized, Role: FOLLOWER
I20260928 23:16:21.274894 22688 consensus_queue.cc:260] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.275213 22688 sys_catalog.cc:564] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:21.275242 22662 raft_consensus.cc:482] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:16:21.275283 22662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:21.276041 22662 raft_consensus.cc:504] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.276300 22662 leader_election.cc:288] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [CANDIDATE]: Term 1 election: Requested vote from peers 898874d060164d16abc206a8f32c0bac (127.0.98.61:38873), e1751526ff57494986a337cee094e77c (127.0.98.60:40591)
I20260928 23:16:21.276721 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f959683b912d483687eb87722d9b323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "898874d060164d16abc206a8f32c0bac"
I20260928 23:16:21.276875 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:21.276846 22812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f959683b912d483687eb87722d9b323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e1751526ff57494986a337cee094e77c"
I20260928 23:16:21.276952 22812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:21.277794 22737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f959683b912d483687eb87722d9b323b in term 1.
I20260928 23:16:21.277794 22812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f959683b912d483687eb87722d9b323b in term 1.
I20260928 23:16:21.278048 22636 leader_election.cc:302] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 898874d060164d16abc206a8f32c0bac, f959683b912d483687eb87722d9b323b; no voters:
I20260928 23:16:21.275179 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.278301 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:21.278402 22872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:21.278477 22872 raft_consensus.cc:686] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [term 1 LEADER]: Becoming Leader. State: Replica: f959683b912d483687eb87722d9b323b, State: Running, Role: LEADER
I20260928 23:16:21.278589 22872 consensus_queue.cc:237] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } }
I20260928 23:16:21.279039 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f959683b912d483687eb87722d9b323b. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.279160 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: This master's current role is: LEADER
W20260928 23:16:21.280191 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:21.280293 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:21.280925 22890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:21.281086 22890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:21.282475 22812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [term 1 FOLLOWER]: Refusing update from remote peer f959683b912d483687eb87722d9b323b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:16:21.282809 22872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:16:21.284070 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f959683b912d483687eb87722d9b323b. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.284147 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:21.284549 22737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [term 1 FOLLOWER]: Refusing update from remote peer f959683b912d483687eb87722d9b323b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:16:21.284816 22872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:16:21.286021 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f959683b912d483687eb87722d9b323b. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.286140 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:21.286940 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.287016 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:21.287369 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.287441 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:21.287698 22892 mvcc.cc:205] Tried to move back new op lower bound from 7334450713731473408 to 7334450713718685696. Current Snapshot: MvccSnapshot[applied={T|T < 7334450713731473408}]
I20260928 23:16:21.288229 22896 mvcc.cc:205] Tried to move back new op lower bound from 7334450713731473408 to 7334450713718685696. Current Snapshot: MvccSnapshot[applied={T|T < 7334450713731473408}]
I20260928 23:16:21.288762 22890 catalog_manager.cc:1347] Generated new cluster ID: 161519ed6dfd49fd91ae1a6e0e0c0b61
I20260928 23:16:21.288811 22890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:21.297765 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.297863 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:21.298110 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f959683b912d483687eb87722d9b323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f959683b912d483687eb87722d9b323b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43629 } } peers { permanent_uuid: "898874d060164d16abc206a8f32c0bac" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38873 } } peers { permanent_uuid: "e1751526ff57494986a337cee094e77c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40591 } } }
I20260928 23:16:21.298200 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:21.298254 22898 mvcc.cc:205] Tried to move back new op lower bound from 7334450713731473408 to 7334450713718685696. Current Snapshot: MvccSnapshot[applied={T|T < 7334450713731473408}]
I20260928 23:16:21.306622 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.281938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.98.60:40591)
0928 23:16:21.281979 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.281981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.285141 (+ 3160us) server_negotiation.cc:407] Connection header received
0928 23:16:21.285279 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.285280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.285290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.285318 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.286170 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.286868 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.293278 (+ 6410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.293569 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.293615 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.301543 (+ 7928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.301544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.301545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.301553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.303228 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.303601 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.303602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.303602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.303657 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.305769 (+ 2112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.305770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.305770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.305907 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.305977 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.306450 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.306529 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:16:21.308135 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.282267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.98.62:43629)
0928 23:16:21.282301 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.282302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.287909 (+ 5607us) server_negotiation.cc:407] Connection header received
0928 23:16:21.287991 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.287992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.288002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.288027 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.297248 (+ 9221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.297941 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.299845 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.300134 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.300174 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.303749 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.303750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.303751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.303757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.305272 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.306185 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.306186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.306186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.306244 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.307331 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.307332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.307333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.307491 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.307579 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.307966 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.308044 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:21.308787 22890 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:21.309511 22890 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:21.318039 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:21.282537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.0.98.61:38873)
0928 23:16:21.282574 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.282576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.293347 (+ 10771us) server_negotiation.cc:407] Connection header received
0928 23:16:21.293442 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.293443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.293453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.293480 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.294185 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.294927 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.296275 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:21.296570 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:21.296606 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:21.315226 (+ 18620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:21.315228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:21.315228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:21.315236 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:21.316816 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.317189 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.317190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.317190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.317259 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:21.317455 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:21.317456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:21.317456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:21.317579 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:21.317651 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:21.317874 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 23:16:21.317947 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:16:21.321285 22890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f959683b912d483687eb87722d9b323b: Generated new TSK 0
I20260928 23:16:21.321480 22890 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:16:21.345069 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.343755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.0.98.60:40591)
0928 23:16:21.343790 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.343792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.344457 (+ 665us) server_negotiation.cc:407] Connection header received
0928 23:16:21.344555 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.344556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.344566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.344594 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.345002 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55214: BlockingRecv error: recv got EOF from 127.0.0.1:55214 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:16:21.346143 22691 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.343561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.98.62:43629)
0928 23:16:21.343604 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.343606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.345596 (+ 1990us) server_negotiation.cc:407] Connection header received
0928 23:16:21.345687 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.345688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.345698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.345726 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.346083 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52742: BlockingRecv error: recv got EOF from 127.0.0.1:52742 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 23:16:21.347004 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.344224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.61:38873)
0928 23:16:21.344263 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.344265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.346484 (+ 2219us) server_negotiation.cc:407] Connection header received
0928 23:16:21.346626 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.346627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.346638 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.346666 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.346943 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37502: BlockingRecv error: recv got EOF from 127.0.0.1:37502 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 23:16:21.349334 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.348589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.0.98.61:38873)
0928 23:16:21.348622 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.348623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.348838 (+ 215us) server_negotiation.cc:407] Connection header received
0928 23:16:21.348932 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.348933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.348948 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.348973 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.349294 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37514: BlockingRecv error: recv got EOF from 127.0.0.1:37514 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:16:21.352152 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.351450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.98.61:38873)
0928 23:16:21.351480 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.351482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.351717 (+ 235us) server_negotiation.cc:407] Connection header received
0928 23:16:21.351804 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.351805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.351813 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.351839 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.352112 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37516: BlockingRecv error: recv got EOF from 127.0.0.1:37516 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 23:16:21.357923 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.355246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.98.61:38873)
0928 23:16:21.355278 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.355279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.357445 (+ 2166us) server_negotiation.cc:407] Connection header received
0928 23:16:21.357537 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.357538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.357547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.357582 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.357880 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37518: BlockingRecv error: recv got EOF from 127.0.0.1:37518 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 23:16:21.363682 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.362963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.0.98.61:38873)
0928 23:16:21.362995 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.362997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.363207 (+ 210us) server_negotiation.cc:407] Connection header received
0928 23:16:21.363291 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.363292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.363301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.363327 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.363643 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37524: BlockingRecv error: recv got EOF from 127.0.0.1:37524 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:16:21.373950 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.373045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.0.98.61:38873)
0928 23:16:21.373085 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.373087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.373381 (+ 294us) server_negotiation.cc:407] Connection header received
0928 23:16:21.373481 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.373482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.373493 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.373523 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.373894 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37538: BlockingRecv error: recv got EOF from 127.0.0.1:37538 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:16:21.391963 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.391301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.0.98.61:38873)
0928 23:16:21.391341 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.391343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.391465 (+ 122us) server_negotiation.cc:407] Connection header received
0928 23:16:21.391514 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.391515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.391526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.391577 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.391901 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37550: BlockingRecv error: recv got EOF from 127.0.0.1:37550 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:16:21.425675 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.424953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.98.61:38873)
0928 23:16:21.425004 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.425006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.425119 (+ 113us) server_negotiation.cc:407] Connection header received
0928 23:16:21.425177 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.425179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.425192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.425257 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.425612 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37558: BlockingRecv error: recv got EOF from 127.0.0.1:37558 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 23:16:21.491317 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.490673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.61:38873)
0928 23:16:21.490738 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.490739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.490815 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:16:21.490864 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.490865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.490876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.490937 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.491252 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37572: BlockingRecv error: recv got EOF from 127.0.0.1:37572 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 23:16:21.620924 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.620336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.0.98.61:38873)
0928 23:16:21.620394 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.620396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.620472 (+ 76us) server_negotiation.cc:407] Connection header received
0928 23:16:21.620522 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.620523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.620533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.620594 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.620866 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37582: BlockingRecv error: recv got EOF from 127.0.0.1:37582 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 23:16:21.878729 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:21.878101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.98.61:38873)
0928 23:16:21.878162 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:21.878164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:21.878174 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:16:21.878239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:21.878240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:21.878250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:21.878302 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:21.878661 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37594: BlockingRecv error: recv got EOF from 127.0.0.1:37594 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 23:16:22.219787 22855 catalog_manager.cc:1259] Loaded cluster ID: 161519ed6dfd49fd91ae1a6e0e0c0b61
I20260928 23:16:22.219875 22855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: loading cluster ID for follower catalog manager: success
I20260928 23:16:22.220647 22855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: acquiring CA information for follower catalog manager: success
I20260928 23:16:22.220930 22855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 898874d060164d16abc206a8f32c0bac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:16:22.235186 22869 catalog_manager.cc:1259] Loaded cluster ID: 161519ed6dfd49fd91ae1a6e0e0c0b61
I20260928 23:16:22.235248 22869 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: loading cluster ID for follower catalog manager: success
I20260928 23:16:22.236042 22869 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: acquiring CA information for follower catalog manager: success
I20260928 23:16:22.236306 22869 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e1751526ff57494986a337cee094e77c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:16:22.394381 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:22.391808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.98.61:38873)
0928 23:16:22.391913 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:22.391915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:22.391976 (+ 61us) server_negotiation.cc:407] Connection header received
0928 23:16:22.392053 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:22.392054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:22.392107 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:22.392176 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:22.392322 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.392842 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.393644 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.393843 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.393898 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:22.394051 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:22.394164 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:22.394231 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:22.394270 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 23:16:22.394291 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260928 23:16:22.398026 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:22.395999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.0.98.60:40591)
0928 23:16:22.396092 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:22.396094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:22.396101 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:16:22.396136 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:22.396137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:22.396170 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:22.396245 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:22.396313 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.396782 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.397462 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.397648 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.397689 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:22.397793 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:22.397889 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:22.397937 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:22.397953 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 23:16:22.397967 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
I20260928 23:16:22.402134 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:22.399568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.0.98.61:38873)
0928 23:16:22.399634 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:22.399635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:22.399730 (+ 95us) server_negotiation.cc:407] Connection header received
0928 23:16:22.399831 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:22.399832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:22.399842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:22.399868 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:22.400005 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.400464 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.401450 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.401662 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.401714 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:22.401906 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:22.401941 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:22.401989 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:22.402039 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 23:16:22.402062 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 23:16:22.402192 22976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:22.399526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.98.60:40591)
0928 23:16:22.399606 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:22.399608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:22.399730 (+ 122us) server_negotiation.cc:407] Connection header received
0928 23:16:22.399773 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:22.399774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:22.399783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:22.399829 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:22.399984 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.400672 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.401488 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.401769 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.401822 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:22.401964 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:22.401994 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:22.402020 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:22.402117 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 23:16:22.402136 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 23:16:22.402649 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:22.399442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.0.98.62:43629)
0928 23:16:22.399598 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:22.399600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:22.399610 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:16:22.399682 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:22.399683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:22.399735 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:22.399817 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:22.399895 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.400656 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.401711 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:22.402001 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:22.402056 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:22.402243 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:22.402352 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:22.402388 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:22.402545 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 23:16:22.402569 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
I20260928 23:16:22.407347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22620
I20260928 23:16:22.412220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22695
I20260928 23:16:22.416239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 22770
2026-09-28T23:16:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2662 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 23:16:22 dist-test-slave-6vgd krb5kdc[22987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 23:16:22 dist-test-slave-6vgd krb5kdc[22987](info): set up 2 sockets
Sep 28 23:16:22 dist-test-slave-6vgd krb5kdc[22987](info): commencing operation
krb5kdc: starting...
W20260928 23:16:23.357192 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/test-user.keytab.
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T23:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T23:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 23:16:23.463721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 23:16:23.541334 23000 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:23.541539 23000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:23.541555 23000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:23.541575 23000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:23.542732 23000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:23.542773 23000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:23.542785 23000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:23.542797 23000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:23.542821 23000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:23.544273 23000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:23.544504 23000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:23.544711 23000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:23.549237 23008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.549259 23005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.549266 23006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:23.549391 23000 server_base.cc:1034] running on GCE node
I20260928 23:16:23.549626 23000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:23.549861 23000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:23.551005 23000 hybrid_clock.cc:648] HybridClock initialized: now 1790637383550993 us; error 28 us; skew 500 ppm
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:23.553294 23000 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 23:16:23.553704 23000 webserver.cc:460] Webserver started at http://127.0.98.62:33487/ using document root <none> and password file <none>
I20260928 23:16:23.553910 23000 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:23.553959 23000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:23.554076 23000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:23.555104 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data/instance:
uuid: "e099c7cfb0964b29b7ddcfd7106541c6"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "243cb8c0ebe8ad15431e43a42bae3d9e"
server_key_iv: "529f3a5be5408181872eebb9ebfdaf2f"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.555430 23000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance:
uuid: "e099c7cfb0964b29b7ddcfd7106541c6"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "243cb8c0ebe8ad15431e43a42bae3d9e"
server_key_iv: "529f3a5be5408181872eebb9ebfdaf2f"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.556618 23000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:23.557296 23015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:23.557438 23000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:23.557519 23000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal
uuid: "e099c7cfb0964b29b7ddcfd7106541c6"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "243cb8c0ebe8ad15431e43a42bae3d9e"
server_key_iv: "529f3a5be5408181872eebb9ebfdaf2f"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.557585 23000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:23.568897 23000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:23.570619 23000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:23.570748 23000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:23.574437 23000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33521
I20260928 23:16:23.574478 23067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33521 every 8 connection(s)
I20260928 23:16:23.574790 23000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/data/info.pb
I20260928 23:16:23.575697 23068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:23.577266 23068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.577966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 23000
I20260928 23:16:23.578060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-0/wal/instance
I20260928 23:16:23.578263 392 external_mini_cluster.cc:1428] Setting key 0e1692eac1c2873f6934698e018417b4
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 23:16:23.586519 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.577928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33521 (local address 127.0.0.1:60990)
0928 23:16:23.578150 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.578190 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.578678 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.578963 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.578965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.579206 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.579366 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.579370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.580347 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.580348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.581027 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.581029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.581708 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.581709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.581774 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.582231 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.582241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.583512 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.585347 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.585353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.585355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.585527 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.585722 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.585722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.585722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.585797 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.586085 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.586088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.586141 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.586209 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.586333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260928 23:16:23.586522 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.577943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.98.62:33521)
0928 23:16:23.578166 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.578176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.578278 (+ 102us) server_negotiation.cc:407] Connection header received
0928 23:16:23.578839 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.578841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.578862 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.578948 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.579458 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.580269 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.581102 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.581612 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.581732 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.583595 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.583604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.583605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.583612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.585266 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.585592 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.585596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.585596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.585672 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.585839 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.585839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.585840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.586037 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.586096 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.586255 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.586403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":187,"spinlock_wait_cycles":7936,"thread_start_us":86,"threads_started":1}
I20260928 23:16:23.587255 23068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.587565 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.587465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35606)
0928 23:16:23.587509 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.587548 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
W20260928 23:16:23.587674 23016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
I20260928 23:16:23.588016 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.587941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35608)
0928 23:16:23.587979 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.588000 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:16:23.588119 23068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 23:16:23.602042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38505
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:16:23.637302 23068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } attempt: 1
I20260928 23:16:23.637800 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.637655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35612)
0928 23:16:23.637715 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.637766 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:16:23.638118 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.638030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35624)
0928 23:16:23.638071 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.638098 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:16:23.638199 23068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
W20260928 23:16:23.680023 23075 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:23.680228 23075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:23.680245 23075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:23.680263 23075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:23.681347 23075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:23.681380 23075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:23.681393 23075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:23.681404 23075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:23.681421 23075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:23.682808 23075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:23.682991 23075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:23.683154 23075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:23.687616 23083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.687625 23080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.687698 23081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:23.687800 23075 server_base.cc:1034] running on GCE node
I20260928 23:16:23.688058 23075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:23.688330 23075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:23.689465 23075 hybrid_clock.cc:648] HybridClock initialized: now 1790637383689451 us; error 26 us; skew 500 ppm
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:23.691803 23075 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 23:16:23.692214 23075 webserver.cc:460] Webserver started at http://127.0.98.61:38777/ using document root <none> and password file <none>
I20260928 23:16:23.692395 23075 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:23.692436 23075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:23.692549 23075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:23.693572 23075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data/instance:
uuid: "3117f626014a4198b363f8e3bcd1cacc"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "1899c9c9f6c286275fd1fcd1b4d13459"
server_key_iv: "1274aebddd571a0e2c29270d99ec4905"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.693883 23075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal/instance:
uuid: "3117f626014a4198b363f8e3bcd1cacc"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "1899c9c9f6c286275fd1fcd1b4d13459"
server_key_iv: "1274aebddd571a0e2c29270d99ec4905"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.695027 23075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 23:16:23.695641 23090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:23.695825 23075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:23.695914 23075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal
uuid: "3117f626014a4198b363f8e3bcd1cacc"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "1899c9c9f6c286275fd1fcd1b4d13459"
server_key_iv: "1274aebddd571a0e2c29270d99ec4905"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.695982 23075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:23.703958 23075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:23.705775 23075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:23.705891 23075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:23.706311 23068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } attempt: 2
I20260928 23:16:23.706634 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.706533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35638)
0928 23:16:23.706583 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.706609 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:16:23.706991 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.706910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35642)
0928 23:16:23.706950 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.706973 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:16:23.707077 23068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38505: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38505: connect: Connection refused (error 111)
I20260928 23:16:23.709542 23075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38505
I20260928 23:16:23.709571 23142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38505 every 8 connection(s)
I20260928 23:16:23.709870 23075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/data/info.pb
I20260928 23:16:23.710763 23143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:23.712447 23143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.716249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 23075
I20260928 23:16:23.716367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-1/wal/instance
I20260928 23:16:23.716553 392 external_mini_cluster.cc:1428] Setting key 32b3e3e3dce8ac0d75fbd6fb9efb1e73
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:23.721081 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.713140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32772 (local address 127.0.98.62:33521)
0928 23:16:23.713180 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.713182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.713433 (+ 251us) server_negotiation.cc:407] Connection header received
0928 23:16:23.713923 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.713925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.713939 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.713989 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.714569 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.715045 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.715996 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.716250 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.716319 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.718569 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.718570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.718570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.718577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.719775 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.720209 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.720210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.720210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.720266 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.720509 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.720509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.720510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.720608 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.720651 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.720960 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.721017 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 23:16:23.721374 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.713112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33521 (local address 127.0.0.1:32772)
0928 23:16:23.713322 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.713357 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.713831 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.714056 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.714059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.714300 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.714486 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.714490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.715130 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.715130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.715919 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.715921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.716376 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.716378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.716526 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.716992 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.716999 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.718481 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.719852 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.719860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.719862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.720139 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.720318 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.720318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.720319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.720452 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.720660 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.720665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.720836 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.720951 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.721106 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":65,"threads_started":1}
I20260928 23:16:23.721907 23143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:23.728837 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.722120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35654)
0928 23:16:23.722179 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.722187 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.722218 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.722424 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.722424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.722544 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.722586 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.722588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.723370 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.723371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.723892 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.723894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.724316 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.724317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.724375 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.724661 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.724664 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.726385 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.727902 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.727903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.727904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.728089 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.728247 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.728248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.728248 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.728302 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.728513 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.728514 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.728557 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.728627 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.728720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 23:16:23.728911 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.722178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.98.61:38505)
0928 23:16:23.722317 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.722320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.722334 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:16:23.722348 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.722350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.722359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.722416 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.722645 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.723304 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.723954 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.724258 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.724313 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.726463 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.726471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.726472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.726479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.727829 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.728154 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.728156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.728157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.728199 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.728349 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.728349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.728350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.728464 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.728516 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.728650 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.728771 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 23:16:23.729542 23143 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.729876 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.729756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38004)
0928 23:16:23.729805 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.729855 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 23:16:23.729998 23091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
I20260928 23:16:23.730381 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.730301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38020)
0928 23:16:23.730341 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.730365 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 23:16:23.730476 23143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37037: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 23:16:23.740228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskawqjjE/build/release/bin/kudu
/tmp/dist-test-taskawqjjE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37037
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 23:16:23.779708 23143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } attempt: 1
I20260928 23:16:23.780206 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.780069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38036)
0928 23:16:23.780121 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.780152 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 23:16:23.780567 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.780490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38044)
0928 23:16:23.780527 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.780551 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 23:16:23.780643 23143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37037: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
I20260928 23:16:23.798220 23068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } attempt: 3
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:23.804924 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.798561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.0.98.61:38505)
0928 23:16:23.798608 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.798610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.798624 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:16:23.798700 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.798701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.798711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.798756 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.799105 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.799625 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.800586 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.800793 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.800841 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.802794 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.802795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.802796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.802804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.803998 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.804283 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.804284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.804284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.804330 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.804481 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.804482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.804482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.804579 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.804629 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.804765 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.804832 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 23:16:23.804924 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.798500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35660)
0928 23:16:23.798559 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.798582 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.798641 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.798795 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.798796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.798970 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.799041 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.799044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.799703 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.799704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.800514 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.800516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.800852 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.800852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.800919 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.801356 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.801362 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.802718 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.804071 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.804072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.804072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.804225 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.804381 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.804382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.804382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.804435 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.804629 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.804630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.804682 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.804738 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.804829 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 23:16:23.805323 23068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.805617 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.805511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38052)
0928 23:16:23.805557 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.805590 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 23:16:23.805960 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.805876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38062)
0928 23:16:23.805914 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.805939 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 23:16:23.806044 23068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37037: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
W20260928 23:16:23.818446 23150 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 23:16:23.818650 23150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 23:16:23.818665 23150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 23:16:23.818684 23150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 23:16:23.819751 23150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 23:16:23.819784 23150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 23:16:23.819799 23150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 23:16:23.819810 23150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 23:16:23.819823 23150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 23:16:23.821269 23150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33521,127.0.98.61:38505,127.0.98.60:37037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 22:19:40 UTC on ccycloud.quasar-blkhea.root.comops.site
build id 4678931
I20260928 23:16:23.821455 23150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 23:16:23.821619 23150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 23:16:23.825958 23158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.825977 23156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 23:16:23.825977 23155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 23:16:23.826182 23150 server_base.cc:1034] running on GCE node
I20260928 23:16:23.826443 23150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 23:16:23.826731 23150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 23:16:23.827881 23150 hybrid_clock.cc:648] HybridClock initialized: now 1790637383827863 us; error 30 us; skew 500 ppm
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 23:16:23.830163 23150 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 23:16:23.830575 23150 webserver.cc:460] Webserver started at http://127.0.98.60:34919/ using document root <none> and password file <none>
I20260928 23:16:23.830762 23150 fs_manager.cc:359] Metadata directory not provided
I20260928 23:16:23.830807 23150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 23:16:23.830924 23150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 23:16:23.831892 23150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data/instance:
uuid: "98ed5530f8284c68b971630e1a4d4708"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "6997c722e6e7a5d3a2fce4d3993e0349"
server_key_iv: "cfaef9d3c65ab871cf928691811da7dd"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.832201 23150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal/instance:
uuid: "98ed5530f8284c68b971630e1a4d4708"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "6997c722e6e7a5d3a2fce4d3993e0349"
server_key_iv: "cfaef9d3c65ab871cf928691811da7dd"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.833402 23150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 23:16:23.834057 23165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 23:16:23.834225 23150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 23:16:23.834321 23150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal
uuid: "98ed5530f8284c68b971630e1a4d4708"
format_stamp: "Formatted at 2026-09-28 23:16:23 on dist-test-slave-6vgd"
server_key: "6997c722e6e7a5d3a2fce4d3993e0349"
server_key_iv: "cfaef9d3c65ab871cf928691811da7dd"
server_key_version: "encryptionkey@0"
I20260928 23:16:23.834388 23150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 23:16:23.837167 23068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } attempt: 1
I20260928 23:16:23.837560 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.837415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38066)
0928 23:16:23.837479 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.837524 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 23:16:23.837867 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.837794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38074)
0928 23:16:23.837827 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.837850 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:16:23.837949 23068 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37037: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
I20260928 23:16:23.848800 23143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } attempt: 2
I20260928 23:16:23.849156 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.849049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38088)
0928 23:16:23.849099 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.849129 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 23:16:23.849416 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.849349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38102)
0928 23:16:23.849380 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.849401 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 23:16:23.849488 23143 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37037: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37037: connect: Connection refused (error 111)
I20260928 23:16:23.863801 23150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 23:16:23.865968 23150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 23:16:23.866083 23150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 23:16:23.869791 23150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37037
I20260928 23:16:23.869832 23217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37037 every 8 connection(s)
I20260928 23:16:23.870167 23150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/data/info.pb
I20260928 23:16:23.871095 23218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 23:16:23.872565 23218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.874431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskawqjjE/build/release/bin/kudu as pid 23150
I20260928 23:16:23.874527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskawqjjE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790637099107577-392-0/minicluster-data/master-2/wal/instance
I20260928 23:16:23.874696 392 external_mini_cluster.cc:1428] Setting key 43bded08cccd8ff988d6cef9b3142963
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:16:23.881902 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.873248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.0.98.62:33521)
0928 23:16:23.873300 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.873302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.873481 (+ 179us) server_negotiation.cc:407] Connection header received
0928 23:16:23.873979 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.873980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.873991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.874042 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.874671 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.875380 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.876580 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.876781 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.876869 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.879122 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.879124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.879124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.879131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.880520 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.880996 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.880997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.880997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.881055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.881329 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.881330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.881330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.881434 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.881480 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.881785 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.881840 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 23:16:23.882153 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.873174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33521 (local address 127.0.0.1:32776)
0928 23:16:23.873361 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.873403 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.873883 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.874113 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.874116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.874366 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.874575 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.874581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.875448 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.875449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.876505 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.876507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.876889 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.876890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.877018 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.877499 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.877508 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.879031 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.880587 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.880593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.880595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.880926 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.881115 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.881115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.881116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.881262 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.881488 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.881493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.881634 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.881764 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.881936 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260928 23:16:23.882649 23218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.883145 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.875022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.0.98.60:37037)
0928 23:16:23.875170 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.875174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.875200 (+ 26us) server_negotiation.cc:407] Connection header received
0928 23:16:23.875229 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.875232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.875249 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.875316 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.875869 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.876812 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.877766 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.878052 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.878105 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.880307 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.880313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.880314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.880319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.881943 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.882305 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.882308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.882308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.882359 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.882546 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.882547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.882547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.882669 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.882947 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.882990 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.883056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 23:16:23.889288 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.882865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38505 (local address 127.0.0.1:35670)
0928 23:16:23.882964 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.882974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.883049 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.883207 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.883208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.883341 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.883394 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.883396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.884177 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.884178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.884876 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.884877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.885315 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.885316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.885367 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.885703 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.885708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.886999 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.888401 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.888402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.888403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.888560 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.888732 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.888733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.888733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.888808 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.889048 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.889049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.889082 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.889129 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.889194 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 23:16:23.889345 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.882924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.98.61:38505)
0928 23:16:23.882963 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.882964 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.883025 (+ 61us) server_negotiation.cc:407] Connection header received
0928 23:16:23.883125 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.883126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.883137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.883201 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.883490 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.884115 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.884951 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.885248 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.885317 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.887072 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.887073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.887073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.887079 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.888318 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.888626 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.888627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.888627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.888678 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.888864 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.888865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.888865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.888991 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.889056 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.889154 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.889254 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:23.889659 23218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } has no permanent_uuid. Determining permanent_uuid...
I20260928 23:16:23.890498 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.883814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.0.98.62:33521)
0928 23:16:23.883851 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.883853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.883859 (+ 6us) server_negotiation.cc:407] Connection header received
0928 23:16:23.883888 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.883889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.883897 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.883937 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.884344 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.885006 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.885710 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.885983 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.886046 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.888026 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.888027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.888027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.888034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.889484 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.889818 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.889819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.889819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.889870 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.890054 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.890055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.890055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.890189 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.890350 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.890389 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.890441 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 23:16:23.895926 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.889859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38124)
0928 23:16:23.889898 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.889907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.889940 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.890056 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.890057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.890216 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.890251 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.890253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.890952 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.890953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.891638 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.891640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.892066 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.892066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.892103 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.892417 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.892422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.893673 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.895026 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.895027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.895027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.895217 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.895402 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.895403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.895403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.895462 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.895670 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.895671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.895711 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.895788 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.895869 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 23:16:23.896023 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.889886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.98.60:37037)
0928 23:16:23.889919 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.889921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.889931 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:16:23.889992 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.889993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.890003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.890050 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.890313 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.890878 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.891713 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.892001 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.892065 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.893748 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.893749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.893749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.893755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.894955 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.895298 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.895298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.895299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.895346 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.895519 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.895520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.895520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.895615 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.895661 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.895812 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.895926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":16}
I20260928 23:16:23.897574 23218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: Bootstrap starting.
I20260928 23:16:23.898272 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.891169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.0.98.61:38505)
0928 23:16:23.891200 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.891201 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.891208 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:16:23.891254 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.891254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.891261 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.891300 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.891771 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.892382 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.893156 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.893475 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.893535 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.895816 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.895817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.895817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.895823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.897259 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.897538 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.897539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.897539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.897601 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.897798 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.897799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.897799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.897920 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.898080 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.898117 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.898192 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 23:16:23.898561 23218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:23.898859 23218 log.cc:824] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:23.899637 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 23:16:23.899694 23218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: No bootstrap required, opened a new log
I20260928 23:16:23.901538 23218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.900723 23042 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e099c7cfb0964b29b7ddcfd7106541c6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790
I20260928 23:16:23.901695 23218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ed5530f8284c68b971630e1a4d4708, State: Initialized, Role: FOLLOWER
I20260928 23:16:23.901785 23218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.902055 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.902101 23218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:23.902144 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:16:23.903774 23235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:23.903826 23235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:23.904074 23150 master_runner.cc:186] Error getting master registration for 127.0.98.62:33521: OK, instance_id { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" instance_seqno: 1790637383573408 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:16:23.906821 23150 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:16:23.913085 23068 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } attempt: 2
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:16:23.920001 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.913419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.60:37037)
0928 23:16:23.913481 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.913483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.913494 (+ 11us) server_negotiation.cc:407] Connection header received
0928 23:16:23.913515 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.913516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.913525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.913582 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.913882 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.914639 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.915580 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.915876 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.915932 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.917915 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.917916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.917916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.917922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.919058 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.919343 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.919343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.919344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.919390 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.919540 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.919541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.919541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.919657 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.919713 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.919850 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.919914 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:23.920002 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.913376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38136)
0928 23:16:23.913430 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.913440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.913475 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.913591 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.913592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.913768 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.913817 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.913819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.914712 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.914713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.915511 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.915513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.915935 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.915936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.915993 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.916426 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.916431 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.917836 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.919129 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.919130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.919130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.919286 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.919439 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.919440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.919440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.919493 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.919721 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.919722 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.919766 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.919823 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.919913 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 23:16:23.921550 23068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: Bootstrap starting.
I20260928 23:16:23.922255 23068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:23.922541 23068 log.cc:824] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:23.923322 23068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: No bootstrap required, opened a new log
I20260928 23:16:23.924635 23068 raft_consensus.cc:348] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.924772 23068 raft_consensus.cc:729] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e099c7cfb0964b29b7ddcfd7106541c6, State: Initialized, Role: FOLLOWER
I20260928 23:16:23.924849 23068 consensus_queue.cc:260] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.925150 23068 sys_catalog.cc:564] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:23.925172 23042 raft_consensus.cc:482] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 23:16:23.925248 23042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:23.925858 23042 raft_consensus.cc:504] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.926076 23042 leader_election.cc:288] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [CANDIDATE]: Term 1 election: Requested vote from peers 3117f626014a4198b363f8e3bcd1cacc (127.0.98.61:38505), 98ed5530f8284c68b971630e1a4d4708 (127.0.98.60:37037)
I20260928 23:16:23.926357 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "98ed5530f8284c68b971630e1a4d4708"
I20260928 23:16:23.926471 23192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:23.927289 23192 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e099c7cfb0964b29b7ddcfd7106541c6 in term 1.
I20260928 23:16:23.927501 23016 leader_election.cc:302] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98ed5530f8284c68b971630e1a4d4708, e099c7cfb0964b29b7ddcfd7106541c6; no voters:
I20260928 23:16:23.925117 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.927953 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:23.927695 23255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 23:16:23.928092 23255 raft_consensus.cc:686] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [term 1 LEADER]: Becoming Leader. State: Replica: e099c7cfb0964b29b7ddcfd7106541c6, State: Running, Role: LEADER
I20260928 23:16:23.928162 23255 consensus_queue.cc:237] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
W20260928 23:16:23.929368 23256 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:23.929416 23256 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:23.929328 23000 master_runner.cc:186] Error getting master registration for 127.0.98.61:38505: OK, instance_id { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" instance_seqno: 1790637383708550 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 23:16:23.931473 23000 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 23:16:23.931531 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e099c7cfb0964b29b7ddcfd7106541c6. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.931614 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:23.931795 23259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 23:16:23.931972 23259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 23:16:23.933586 23192 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [term 1 FOLLOWER]: Refusing update from remote peer e099c7cfb0964b29b7ddcfd7106541c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:16:23.933934 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:16:23.935874 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e099c7cfb0964b29b7ddcfd7106541c6. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.935966 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:23.936813 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.936892 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:23.937191 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.937285 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:23.937500 23261 mvcc.cc:205] Tried to move back new op lower bound from 7334450724590280704 to 7334450724571041792. Current Snapshot: MvccSnapshot[applied={T|T < 7334450724590280704}]
I20260928 23:16:23.938004 23262 mvcc.cc:205] Tried to move back new op lower bound from 7334450724590280704 to 7334450724571041792. Current Snapshot: MvccSnapshot[applied={T|T < 7334450724590280704}]
I20260928 23:16:23.940770 23143 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } attempt: 3
I20260928 23:16:23.941495 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.927427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.0.98.61:38505)
0928 23:16:23.927556 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.927607 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.932238 (+ 4631us) server_negotiation.cc:407] Connection header received
0928 23:16:23.932329 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.932330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.932340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.932371 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.933011 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.933717 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.935545 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.936254 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.936299 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.938234 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.938235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.938235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.938242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.939759 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.940156 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.940157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.940157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.940232 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.940514 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.940515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.940515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.940678 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.940751 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.941329 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.941409 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 23:16:23.942090 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.927382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.0.98.60:37037)
0928 23:16:23.927492 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.927494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.933795 (+ 6301us) server_negotiation.cc:407] Connection header received
0928 23:16:23.933815 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.933816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.933825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.933866 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.934526 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.935194 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.937247 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.937538 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.937580 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.939336 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.939338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.939338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.939345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.940807 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.941170 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.941171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.941172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.941253 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.941552 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.941554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.941554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.941689 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.941776 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.941940 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.942011 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 23:16:23.943150 23259 catalog_manager.cc:1347] Generated new cluster ID: b812f5f319d24452b206cc9b4c492d4f
I20260928 23:16:23.943197 23259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 23:16:23.947065 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.931961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.98.62:33521)
0928 23:16:23.932001 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.932003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.933803 (+ 1800us) server_negotiation.cc:407] Connection header received
0928 23:16:23.934566 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.934567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.934577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.934605 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.941269 (+ 6664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.941901 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.942763 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.943006 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.943060 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.944361 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.944362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.944362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.944368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.945798 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.946203 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.946204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.946205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.946251 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.946480 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.946481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.946481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.946600 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.946662 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.946907 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.946985 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 23:16:23 dist-test-slave-6vgd krb5kdc[22987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790637383, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 23:16:23.958422 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.940960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37037 (local address 127.0.0.1:38144)
0928 23:16:23.941110 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 23:16:23.941120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 23:16:23.941155 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 23:16:23.941704 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 23:16:23.941706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 23:16:23.941845 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 23:16:23.941884 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.941886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.942974 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.942975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.943680 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 23:16:23.943682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.953649 (+ 9967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.953651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 23:16:23.953733 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.954082 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 23:16:23.954085 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 23:16:23.955419 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 23:16:23.957290 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.957290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.957291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.957475 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.957756 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 23:16:23.957756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 23:16:23.957757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 23:16:23.957817 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 23:16:23.958151 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 23:16:23.958151 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 23:16:23.958202 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 23:16:23.958248 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 23:16:23.958331 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 23:16:23.959085 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:23.940997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.98.60:37037)
0928 23:16:23.941576 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:23.941577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:23.941584 (+ 7us) server_negotiation.cc:407] Connection header received
0928 23:16:23.941600 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:23.941601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:23.941610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:23.941945 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:23.941990 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.942911 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.953246 (+ 10335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:23.953583 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:23.953640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:23.955510 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 23:16:23.955511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 23:16:23.955512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 23:16:23.955518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 23:16:23.957190 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.957558 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.957559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.957560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.957645 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 23:16:23.957878 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 23:16:23.957879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 23:16:23.957879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 23:16:23.958071 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 23:16:23.958691 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:23.958774 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 23:16:23.958894 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":79,"threads_started":1}
I20260928 23:16:23.960860 23143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: Bootstrap starting.
I20260928 23:16:23.961629 23143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: Neither blocks nor log segments found. Creating new log.
I20260928 23:16:23.961931 23143 log.cc:824] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: Log is configured to *not* fsync() on all Append() calls
I20260928 23:16:23.963106 23143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: No bootstrap required, opened a new log
I20260928 23:16:23.965225 23143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.965370 23143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3117f626014a4198b363f8e3bcd1cacc, State: Initialized, Role: FOLLOWER
I20260928 23:16:23.965452 23143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } }
I20260928 23:16:23.965764 23143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 23:16:23.965842 23116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [term 0 FOLLOWER]: Advancing to term 1
I20260928 23:16:23.965845 23117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3117f626014a4198b363f8e3bcd1cacc"
I20260928 23:16:23.966675 23116 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [term 1 FOLLOWER]: Refusing update from remote peer e099c7cfb0964b29b7ddcfd7106541c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 23:16:23.966984 23255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 23:16:23.967484 23117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e099c7cfb0964b29b7ddcfd7106541c6 in term 1.
I20260928 23:16:23.968122 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.968207 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: This master's current role is: FOLLOWER
W20260928 23:16:23.968317 23279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 23:16:23.968401 23279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 23:16:23.970287 23259 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 23:16:23.970440 23238 sys_catalog.cc:455] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.970530 23238 sys_catalog.cc:458] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6 [sys.catalog]: This master's current role is: LEADER
I20260928 23:16:23.970953 23259 catalog_manager.cc:1504] Loading token signing keys...
I20260928 23:16:23.973295 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e099c7cfb0964b29b7ddcfd7106541c6. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.973433 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:23.973492 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e099c7cfb0964b29b7ddcfd7106541c6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33521 } } peers { permanent_uuid: "3117f626014a4198b363f8e3bcd1cacc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38505 } } peers { permanent_uuid: "98ed5530f8284c68b971630e1a4d4708" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37037 } } }
I20260928 23:16:23.973567 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc [sys.catalog]: This master's current role is: FOLLOWER
I20260928 23:16:23.981285 23259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e099c7cfb0964b29b7ddcfd7106541c6: Generated new TSK 0
I20260928 23:16:23.981459 23259 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 23:16:24.004130 23071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.002573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.0.98.62:33521)
0928 23:16:24.002616 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.002619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.003484 (+ 865us) server_negotiation.cc:407] Connection header received
0928 23:16:24.003582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.003583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.003593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.003622 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.004060 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32800: BlockingRecv error: recv got EOF from 127.0.0.1:32800 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 23:16:24.004835 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.002958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.61:38505)
0928 23:16:24.002994 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.002996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.004353 (+ 1357us) server_negotiation.cc:407] Connection header received
0928 23:16:24.004443 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.004444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.004453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.004479 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.004761 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35696: BlockingRecv error: recv got EOF from 127.0.0.1:35696 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 23:16:24.005544 23266 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.003258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.98.60:37037)
0928 23:16:24.003295 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.003297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.005065 (+ 1768us) server_negotiation.cc:407] Connection header received
0928 23:16:24.005159 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.005160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.005170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.005198 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.005487 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38154: BlockingRecv error: recv got EOF from 127.0.0.1:38154 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 23:16:24.007628 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.006962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.0.98.61:38505)
0928 23:16:24.006999 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.007001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.007173 (+ 172us) server_negotiation.cc:407] Connection header received
0928 23:16:24.007267 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.007268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.007283 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.007308 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.007586 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35702: BlockingRecv error: recv got EOF from 127.0.0.1:35702 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 23:16:24.010439 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.009789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.0.98.61:38505)
0928 23:16:24.009818 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.009820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.010001 (+ 181us) server_negotiation.cc:407] Connection header received
0928 23:16:24.010097 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.010098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.010107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.010132 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.010398 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35712: BlockingRecv error: recv got EOF from 127.0.0.1:35712 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:16:24.014171 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.013513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.98.61:38505)
0928 23:16:24.013544 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.013546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.013729 (+ 183us) server_negotiation.cc:407] Connection header received
0928 23:16:24.013812 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.013813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.013822 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.013847 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.014133 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35714: BlockingRecv error: recv got EOF from 127.0.0.1:35714 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 23:16:24.019804 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.019173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.0.98.61:38505)
0928 23:16:24.019203 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.019205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.019380 (+ 175us) server_negotiation.cc:407] Connection header received
0928 23:16:24.019462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.019462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.019472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.019498 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.019765 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35716: BlockingRecv error: recv got EOF from 127.0.0.1:35716 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:16:24.029681 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.028932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.0.98.61:38505)
0928 23:16:24.028976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.028978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.029188 (+ 210us) server_negotiation.cc:407] Connection header received
0928 23:16:24.029301 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.029302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.029311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.029338 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.029636 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35730: BlockingRecv error: recv got EOF from 127.0.0.1:35730 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 23:16:24.047696 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.046885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.0.98.61:38505)
0928 23:16:24.046920 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.046922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.047165 (+ 243us) server_negotiation.cc:407] Connection header received
0928 23:16:24.047261 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.047262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.047273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.047300 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.047647 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35740: BlockingRecv error: recv got EOF from 127.0.0.1:35740 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 23:16:24.081943 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.081011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.98.61:38505)
0928 23:16:24.081072 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.081074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.081359 (+ 285us) server_negotiation.cc:407] Connection header received
0928 23:16:24.081459 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.081460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.081471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.081499 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.081881 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35756: BlockingRecv error: recv got EOF from 127.0.0.1:35756 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 23:16:24.147852 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.147184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.0.98.61:38505)
0928 23:16:24.147236 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.147238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.147345 (+ 107us) server_negotiation.cc:407] Connection header received
0928 23:16:24.147408 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.147408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.147420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.147466 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.147791 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35772: BlockingRecv error: recv got EOF from 127.0.0.1:35772 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 23:16:24.277688 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.276943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.0.98.61:38505)
0928 23:16:24.277021 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.277023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.277095 (+ 72us) server_negotiation.cc:407] Connection header received
0928 23:16:24.277192 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.277193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.277223 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.277287 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.277622 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35784: BlockingRecv error: recv got EOF from 127.0.0.1:35784 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 23:16:24.535391 23147 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 23:16:24.534831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.98.61:38505)
0928 23:16:24.534890 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:24.534891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:24.534905 (+ 14us) server_negotiation.cc:407] Connection header received
0928 23:16:24.534979 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:24.534980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:24.534990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:24.535040 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 23:16:24.535342 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35794: BlockingRecv error: recv got EOF from 127.0.0.1:35794 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 23:16:24.904232 23235 catalog_manager.cc:1259] Loaded cluster ID: b812f5f319d24452b206cc9b4c492d4f
I20260928 23:16:24.904310 23235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: loading cluster ID for follower catalog manager: success
I20260928 23:16:24.905107 23235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: acquiring CA information for follower catalog manager: success
I20260928 23:16:24.905435 23235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98ed5530f8284c68b971630e1a4d4708: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:16:24.968870 23279 catalog_manager.cc:1259] Loaded cluster ID: b812f5f319d24452b206cc9b4c492d4f
I20260928 23:16:24.968946 23279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: loading cluster ID for follower catalog manager: success
I20260928 23:16:24.969715 23279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: acquiring CA information for follower catalog manager: success
I20260928 23:16:24.969980 23279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3117f626014a4198b363f8e3bcd1cacc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 23:16:25.050745 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:25.048469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.0.98.61:38505)
0928 23:16:25.048566 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:25.048568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:25.048578 (+ 10us) server_negotiation.cc:407] Connection header received
0928 23:16:25.048652 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:25.048653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:25.048706 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:25.048773 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:25.048891 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.049436 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.050131 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.050321 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.050364 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:25.050472 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:25.050575 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:25.050626 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:25.050642 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 23:16:25.050658 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260928 23:16:25.054560 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:25.052329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.0.98.60:37037)
0928 23:16:25.052453 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:25.052455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:25.052463 (+ 8us) server_negotiation.cc:407] Connection header received
0928 23:16:25.052496 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:25.052497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:25.052532 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:25.052597 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:25.052692 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.053171 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.053888 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.054078 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.054130 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:25.054260 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:25.054357 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:25.054419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:25.054455 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 23:16:25.054476 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":46,"threads_started":1}
I20260928 23:16:25.058486 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:25.055882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.98.61:38505)
0928 23:16:25.055918 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:25.055920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:25.056090 (+ 170us) server_negotiation.cc:407] Connection header received
0928 23:16:25.056147 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:25.056148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:25.056158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:25.056207 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:25.056325 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.056844 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.057734 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.058010 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.058113 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:25.058251 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:25.058279 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:25.058324 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:25.058392 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 23:16:25.058412 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 23:16:25.058887 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:25.056070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.98.60:37037)
0928 23:16:25.056101 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:25.056103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:25.056258 (+ 155us) server_negotiation.cc:407] Connection header received
0928 23:16:25.056279 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:25.056280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:25.056289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:25.056329 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:25.056451 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.057143 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.058022 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.058294 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.058355 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:25.058593 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:25.058626 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:25.058663 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:25.058803 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 23:16:25.058824 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 23:16:25.059172 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 23:16:25.055929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.0.98.62:33521)
0928 23:16:25.056045 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 23:16:25.056047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 23:16:25.056168 (+ 121us) server_negotiation.cc:407] Connection header received
0928 23:16:25.056210 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 23:16:25.056211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 23:16:25.056256 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 23:16:25.056334 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 23:16:25.056487 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.057245 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.058049 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 23:16:25.058305 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 23:16:25.058359 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 23:16:25.058679 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 23:16:25.058778 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 23:16:25.058846 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 23:16:25.058898 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 23:16:25.058919 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":49,"threads_started":1}
I20260928 23:16:25.060623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 23000
I20260928 23:16:25.066282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 23075
I20260928 23:16:25.070900 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskawqjjE/build/release/bin/kudu with pid 23150
2026-09-28T23:16:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2654 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5317 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285968 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS